Commit Graph

43 Commits

Author SHA1 Message Date
Teknium fc8d15d779 test: retain a reproducible CLI goal parity oracle 2026-09-07 00:51:41 -07:00
Teknium 3be1b87af1 fix(gateway): keep heartbeat admission on its watched route 2026-09-07 00:34:12 -07:00
Teknium be58c276ee feat(compression): per-image token cost learned from the provider's own usage (#70328, supersedes #70463)
A flat per-image constant (1500 in the trigger estimator, 1600 in the tail-budget walk) is wrong in
both directions: a screenshot costs ~1,100 tokens on one provider and 4,000+ on a local mmproj
model. In a GUI loop on a 64K window the estimate sat at ~20K while the real prompt passed 80K,
so compaction never fired and the provider rejected every request (#70328).

The provider prices every image exactly on the request that carries it, so the cost is
observable from usage alone, with no vendor formula: with a fresh usage anchor, the residual
between the next real prompt_tokens and anchor + text-only delta is the price of the N images
that delta introduced.

- agent/image_token_cost.py: calibrate_from_usage() runs in record_response_usage before the new
  anchor is captured; the learned value (EMA, plausibility-banded) is kept per model@host in
  ~/.hermes/cache/image_token_costs.json and bound per turn through a ContextVar.
- estimate_messages_tokens_rough, _content_length_for_budget (tail walk) and gateway hygiene all
  read the same bound value, so trigger and walk agree; the per-message memo now caches text
  tokens and image COUNT so a recalibration re-prices cached rows.
- One flat default (1500) remains only until the first vision turn; the duplicate 1600 is gone.

evals/token_accounting/ab_image_cost_calibration.py (real AIAgent, fake provider pricing images
at 4,000, one screenshot per turn, 64K window): main learns nothing (1500) and the tail walk
under-prices its own protected tail by 56.5%; this branch learns 4,374 after one vision turn
and the walk's error is +8.5%.

Reporter and first-fix credit: @JonthanaHanh (#70328, #70463).
2026-09-06 14:19:42 -07:00
Teknium 6d2645e64e feat(logging): the API call line carries cache write count, the provider's response id, and the serving upstream
Diagnosing the 1,393-agent run's cache misses took a state.db join and a live
probe because none of the three were on the one line we log per call:

  write=<n>    cache_creation tokens; a write costs 50x a read, so this is the
               money, and "read stuck, write large" on consecutive lines is a
               routing miss visible without a probe
  id=<id>      the provider's response id (Anthropic msg_..., OpenRouter/Nous
               gen-...): what a provider needs to look a request up
  upstream=<n> who actually served, when the route reports it (OpenRouter's
               `provider`); how we learned GMI was not serving

Fields are appended to the existing line and omitted when absent, so every
existing parser (evals/postmortem, the two cache_prefix probes) keeps
matching; the forensics parser reads them when present.

The streamed chat path fabricated `id="stream-<uuid>"` and dropped the
chunks' id/provider on the floor; it now keeps the first chunk's id and
provider, falling back to the fabricated id only when the stream never sent
one. Nothing depended on the prefix.

Live (Nous, Fable 5.1, both wires): chat -> `write=3759
id=gen-1788727882-... upstream=Anthropic`; native -> `id=gen-1788727891-...`
(the Anthropic Message object's id was already real). Tests: fields present
and omitted, prefix unchanged; forensics parser reads new and old lines.
2026-09-06 13:52:15 -07:00
Teknium 1d06f7a95a docs+evals(compression): token-accounting replay harness and developer-guide section (#104462)
evals/token_accounting/replay_gates.py runs the REAL AIAgent turn loop against a local fake
chat-completions server with scripted usage.prompt_tokens, in three shapes (CLI same-object
history, gateway JSON-reloaded history, SessionDB close/reopen + fresh agent) x two arms
(transcript 3x threshold by bytes/4 while real usage is under; transcript tiny while real usage is
over). Acceptance: no gate fires when real usage is under threshold regardless of estimate
inflation; every gate fires once real usage is over.

origin/main 5f406d88ea: cli/gateway/restore inflated all FAIL (1 local compaction each on the
estimate). This branch: 6/6 PASS, anchor restored in the fresh process.
2026-09-06 13:21:17 -07:00
Teknium 8f2167342f evals(postmortem): notice probes target a grouped unit (per-group split on main; ungrouped tasks already report individually) 2026-09-06 12:12:09 -07:00
Teknium 583fb3e200 evals(postmortem): cache concurrency probe, the instrument behind #104284/#104421 and api#227
N concurrent real AIAgent sessions on a growing tool loop; per-call cache_read /
cache_creation / response id / upstream / prefix shas; consecutive pairs classified
ideal / stuck / collapse. argparse (--provider nous|openrouter|anthropic, --wire,
--pin, --settle, --ttl, --model); wire defaults to what Hermes would pick. Summary
JSON carries bad pairs with both response ids. Smoke-run against Nous from the repo
path (picked chat via nous_api_mode; 6/6 ideal).
2026-09-06 10:23:38 -07:00
Teknium 1690a7a11f Merge remote-tracking branch 'origin/main' into feat/evals-postmortem-harness 2026-09-06 10:20:50 -07:00
Teknium 8d4b7f874d eval(native-compaction): real-AIAgent A/B probe for checkpoint preflight false-trigger
Fake local Responses SSE server plays the Codex backend; scenarios cover live
capture (CLI same-object history and gateway reloaded history), SessionDB
close/reopen restore into a fresh agent, and the over-threshold negative where
local compression must still fire after real usage arrives.
2026-09-06 09:09:00 -07:00
Teknium 7166071fca fix(auxiliary): ResourceExhausted spellings reach the payment/quota fallback gate
The compression/aux fallback ladder decides "payment error" via
_is_payment_error(), which only knew the spaced "resource exhausted". NVIDIA NIM
and gRPC-style wrappers serialize the same quota signal as ResourceExhausted /
RESOURCE_EXHAUSTED / resource-exhausted, so a 403/429/status-less body carrying
it re-raised instead of walking the configured fallback_chain, and compression
fell to the lossy emergency path. Add the three separator variants to
_PAYMENT_KEYWORDS (same status gate, no broader "exhausted" matching).

evals/auxiliary_resource_exhausted.py drives the real call_llm/async_call_llm
against two local OpenAI-compatible listeners (nvidia profile primary, named
custom fallback) for the before/after check. #85649
2026-09-06 05:48:39 -07:00
Teknium 91b4d9fb67 test(fallback): keep the two persisted-config invariants; PTY harness for the picker error path
Trim the salvaged suite to the behaviour contracts: a picker exception or Ctrl+C leaves
config.yaml's model exactly as snapshotted (parametrized), and an absent active_provider
stays absent through snapshot+restore. The mock-dispatch tests (asserting calls into our own
helpers) are dropped. evals/cli_fallback_add_picker_error.py drives the real
`hermes fallback add` under a Linux PTY into an ordinary picker OSError and checks the
persisted model: stranded on base, restored after.
2026-09-06 05:41:46 -07:00
Teknium e0592b6d43 fix(codex): masked "invalid_prompt: Request blocked." replay rejection reaches the replay-strip recovery
The ChatGPT Codex backend answers a rejected encrypted-reasoning replay with the
same bare 400 {code: invalid_prompt, message: "Request blocked."} it uses for
genuine blocks. classify_api_error bucketed it as format_error, so the one-shot
repair that already exists for invalid_encrypted_content (disable replay, strip
cached codex_reasoning_items, retry once) never ran and the turn aborted.

Classify exactly that envelope from provider openai-codex — SDK 400 body, SSE
``error`` frame, or the ``response.failed`` text — as invalid_encrypted_content
while keeping format_error's abort-and-fallback hints, so the only behavioural
delta is turn_recovery's replay strip, which still requires cached reasoning
items. A block with nothing to strip aborts exactly as before; other providers,
other messages/codes and the #18028 safety refusals are untouched.

evals/codex_masked_replay_review.py drives a real AIAgent (codex_responses)
against a synthetic local Responses server for the before/after check.
Refs #92353, #92357, #94077.
2026-09-06 05:40:02 -07:00
Teknium ebb428aed5 test(anthropic): trim DeepSeek proxy replay tests to two invariants + local wire probe
Fold the adapter-level unsigned-preservation case and the name-detector matrices
into one parametrized invariant (thinking family keeps unsigned only; non-thinking
DeepSeek and unrelated models still take the generic third-party strip) plus the
native-endpoint immutability guard. Add evals/anthropic_proxy_thinking_replay.py,
a real-AIAgent probe against a synthetic local Anthropic-compatible relay used for
the before/after wire check.
2026-09-06 05:37:03 -07:00
Teknium 3a15c39e8e fix(banner): deferred update notice keeps prompt_toolkit routing; drop dead console arg; PTY harness
Follow-up on the cherry-picked fix: _defer_update_notice no longer takes the Rich console it
can never safely use (the notice always lands after patch_stdout owns stdout), the unit test
asserts the contract at prompt_toolkit's boundary (an ANSI fragment reaches
print_formatted_text, with no raw ESC/markup in the visible text) instead of patching our own
cprint, and evals/cli_deferred_notice.py drives the real CLI under a Linux PTY with a FIFO-gated
update cache to A/B the late-notice cases (garbled on base, clean after).
2026-09-06 05:36:26 -07:00
Teknium e40192118e fix(webhook): only webhook-signature commits a delivery to the Standard Webhooks path
GitLab (lib/gitlab/web_hooks.rb, app/services/web_hook_service.rb) sends
webhook-id and webhook-timestamp on EVERY delivery and adds webhook-signature
only when a signing token is configured. Selecting the HMAC path on any
webhook-* header would 401 every legacy X-Gitlab-Token install the moment it
upgrades to GitLab 19, so only the signature header selects the path; id and
timestamp then travel with it and the validator still fails closed when either
is missing. svix-* keeps its existing any-header selection.

Tests trimmed to the invariant bar: one parameterized contract (whsec_ and raw
secrets accept; wrong secret, tampered body and stale timestamp reject) plus the
GitLab legacy-token coexistence contract. evals/webhook_auth/standard_webhooks_ab.py
drives a real aiohttp WebhookAdapter on a dedicated loopback port with real
signed requests for before/after evidence.

Related: #47849 (HwangJohn, cherry-picked here), #92024 (earlier salvage of
#47849), #102080 and #103167 (same alias fix, same target).
2026-09-06 05:35:48 -07:00
Teknium 62f353eed5 refactor(gateway): iteration-progress formatter lives in agent/session_activity, tests trimmed to the invariant bar
The salvaged helper was appended to the gateway/run.py facade; new behaviour
belongs in a topical sibling. agent/session_activity.py already owns the
activity-snapshot contract the three render sites read from, so the formatter
moves there as format_iteration_progress and the three call sites import it at
module level instead of late-importing the facade.

Tests trimmed to the salvage bar (<= 2 invariant tests per fix): one
parameterized contract on the helper (unbounded/None hide the ceiling, a real
budget keeps N/M) plus the contributor's end-to-end busy-ack test through the
real render path. The per-site heartbeat/timeout tests exercised the same
helper through mocks and are dropped; evals/gateway_status_render/
iteration_ceiling_ab.py drives all three real render sites with a real AIAgent
for before/after evidence (3 sentinel leaks on main -> 0).

Related: #103109 (same fix, same target module; credited in the PR),
#102845 (third render site, closed by its author in favour of #102817).
2026-09-06 05:35:12 -07:00
Teknium 4c287fa7bf test(evals): wire-contract A/B harness for Slack chat.*Stream payloads
Replays the native task-card rail and the text-stream rail through the real
slack_sdk AsyncWebClient against a local aiohttp receiver that enforces the
documented markdown_text/chunks mutual exclusion. Local contract proof, not
Slack live: records outgoing bodies before/after #87743.
2026-09-06 05:34:36 -07:00
Teknium a8ca904922 feat(evals): post-mortem harness — forensics lanes + live A/B + review probes for the #102117 run fixes
evals/postmortem/ turns the one-off audit behind tracking issue #103563 into
something anyone with a Hermes state.db copy (and optionally rotated
agent.log*) can run on their own fan-out:

  forensics/   common.py discovers the run tree (root = most descendants,
               compression-rollover children excluded so cost buckets stay
               disjoint), fits pricing from estimated_cost_usd, and five lanes
               recompute the OBSERVED figures: tokens (buckets, depth/duration
               shares, context reconstruction, excess-cache-write proxy, cap
               replay), logcalls (per-call cache behaviour from agent.log with
               coverage printed first; strict and loose plateau definitions
               reported separately), delegation (timeouts, orphaned children,
               polling hours, batch-join withheld child-hours, truncated
               summaries), tools (hardline blocks, foreground refusals,
               whole-file rewrites), goal_loop (nudges, parked barrier), rework
               (public-surface drop at PR open + post-open commit inventory).
               Every figure is labeled OBSERVED or MODELED.
  live_ab/     the per-PR A/Bs (real code paths, fake providers, temp
               HERMES_HOME), paths from argv.
  review_probes/ the independent /review's probes, credited and adapted; each
               reproduced a round-1 defect and the fixed head must pass it.
  run.py       runs the offline probes against one or two checkouts and prints
               PASS/FAIL side by side (--live adds the ones that spend cents).
  tests/       synthetic-DB smoke test for the lanes and runner.

On the run's DB the lanes reproduce the tracking issue's population exactly
(1,394 sessions, 93,284 calls, $19,302.59; cache_write $11,159.76) and on
main vs an integration checkout of the 13 PRs the runner shows every probe
FAIL -> PASS (two guard-only probes pass on both, noted in run.py).

The trajectories are deliberately not shipped: the DB holds 51,956 home
paths, 5,341 e-mails, private IPs, chat ids and real-shaped credentials in
tool output. The lane reports and recomputed JSON are in a secret gist
linked from #103563.
2026-09-05 09:13:10 -07:00
Teknium 27a4023791 docs+evals: AGENTS.md for the facade/siblings layout; codebase-navigability benchmark harness
AGENTS.md: Project Structure tree reflects the decomposition (run_agent 1.5k not 12k, cli 4.6k not 11k,
hermes_state facade + 21 siblings, web_routers/, evals/, test counts); new "Facade + siblings layout"
section with the sibling families table and the rules that follow (find by topic, patch where production
reads, compat pointers off limits, don't recreate god files); AIAgent/Agent Loop point at agent/turn_*.py
and conversation_loop; CLI dispatch documents _SLASH_DISPATCH + the _handle_<name>_command convention and
"Adding a Slash Command" no longer tells you to add an elif (there is no ladder to add to on either surface).

evals/codebase_navigability/: what the codebase costs an agent, not the CPU.
  bench.py         ~19k real "locate X" tasks from tests/ imports; tokens (tiktoken o200k) of the defining
                   file vs the symbol, context-window fit, read windows, siblings, symbol CC
  lookup_sim.py    paired grep+read simulation over 4k common symbols; tool calls + tokens returned
  static_metrics.py LOC split, size distributions, elif/nesting, radon CC/MI, import graph + SCC cycles
  runtime_bench.py  fresh-interpreter import/CLI/hot-path/collection timings with tree-purity assertion
tests/evals/test_codebase_navigability.py pins the resolver's facade/sibling behaviour.
2026-09-03 23:55:56 -07:00
Teknium 9cee679831 bench: explicit utf-8 encoding on text-mode opens (ruff PLW1514) 2026-09-03 02:31:59 -07:00
Teknium 45b0a0ae25 bench: --reply-kb payload padding, transport/live-agent gc counts, after-snapshot rows in --compare 2026-09-03 02:31:59 -07:00
Teknium c77b9d637b bench: fan-out resource harness (threads/RSS/fds/pyright/kernels/httpx clients per N children x W worktrees) 2026-09-03 02:31:59 -07:00
Teknium 6101f52ba4 Merge remote-tracking branch 'origin/main' into core-tool-deferral 2026-08-30 19:47:30 -07:00
Teknium 4f22543509 fix(compression): lean compaction makes exactly one auxiliary request per attempt
The lean tail mode's per-chunk digest loop (_build_chunk_digests) issued up
to 28 extra call_llm requests sequentially per compaction attempt. With lean
now the default (#95571), users on slow auxiliary routes hit 7-11 minute
compactions (#96603). Remove the loop entirely: a lean compaction attempt now
makes EXACTLY ONE auxiliary LLM request — the main summary call.

- The detailed session log is folded into the single summary request: the
  lean prompt template gains a '## Detailed Session Log (oldest first)'
  section carrying the digest prompt's HARD RULES (identifiers verbatim,
  dense bullets, transcript-is-data). Output guidance grows by
  _LEAN_SESSION_LOG_BUDGET_TOKENS = 4,000 tokens on top of the scaled
  summary budget — the old worst case (28 x 1,400 digest tokens) was spread
  across many requests and mostly re-covered tool noise; a single dense
  4K-token log inside one response preserves the load-bearing record while
  staying well inside one aux response (the summary call still sends no hard
  max_tokens, so no provider cap can truncate it mid-section).
- Input sizing: oversized regions (500K+ chars) are EVEN-SAMPLED across the
  whole region (_sample_summary_input: 8 proportionally spaced slices,
  oldest-to-newest, explicit '[... N chars elided ...]' markers, last slice
  anchored to the newest end) instead of head+tail truncated, so session-log
  coverage stays uniform. Legacy mode keeps _bound_summary_input unchanged.
- The LLM-free anchor index still runs over the FULL region, and the
  session_search recovery footer is unchanged.
- Dead code removed: _build_chunk_digests, _LEAN_DIGEST_* constants,
  _LEAN_DIGEST_PROMPT, _serialize_turns_for_digest, _digest_worthy,
  _LOW_SIGNAL_TOOL_RE, the _lean_pristine_tools snapshot, and the
  sibling-call route echo (_SUMMARY_ROUTE_CONSUMED /
  attempt_summary_route_kwargs — no remaining callers; the single-use
  summary pin semantics are unchanged).
- Tests pin the new contract (exactly one call_llm in lean mode; session-log
  section lands in the summary; oversized regions sampled with elision
  markers, never a second request; anchor index + recovery footer present).
  Sabotage-verified: restoring a second call_llm makes the call-count test
  fail. Docs and the compaction eval wording updated to stop claiming
  per-chunk calls.

Fixes #96603.
2026-08-30 09:03:57 -07:00
Teknium 5bbb4cd6a9 lint: explicit encoding on harness file opens (ruff unspecified-encoding) 2026-08-29 19:04:29 -07:00
Teknium 037ce5cf75 eval(tool-search): check in the core-tool-deferral live A/B harness
The harness behind this PR's 288-run maintainer battery, ported from /tmp
into evals/ alongside the readtool / session_search_schema harnesses.
Arm trees parametrized via ABDEFER_BASE_TREE / ABDEFER_PR_TREE (pinned
plain checkouts), results/python roots via ABDEFER_RESULTS / ABDEFER_PYTHON.

- tasks.py: 14 tasks — per-deferred-tool coverage, multistep, long-range,
  clarify ambiguity trap, eager-only control, false-discovery distractor;
  programmatic graders with partial credit
- worker.py: isolated per-cell subprocess, temp HERMES_HOME, hermetic env,
  seeded session DB with decoys, deterministic desktop/computer_use/
  image_generate stubs, interactivity-fairness continuation, exit-3
  infra-abort (misconfig never scores)
- orchestrator.py: resume-safe battery runner, wall timeouts, retry of
  errored records, infra-abort fuse
- report.py: per-task A/B tables + mean-of-task-means
- results/SUMMARY.md: the shipped verdict; rep JSONs gitignored

Ported worker re-verified live post-port (real terra cell, score 1.0).
2026-08-29 18:42:59 -07:00
Teknium dfff42d3dd refactor(browser_exec): schema diet 803->663 tok/call, A/B-gated; eval harness runs on Windows + Nous auth (#96300) 2026-08-27 04:39:19 -07:00
Teknium b2bd1ac63f eval: session_search schema A/B harness + PR #95570 reference results
Live tool-use A/B for session_search schema changes: arms are git refs
(tools/session_search_tool.py extracted per ref), tasks run a minimal
agent loop over OpenRouter against a freshly seeded temp session DB with
programmatic oracles — discovery, forced forward-scroll, AND-miss
broadening, verbatim link emission, profile-link resolution, browse.

Checked-in results/pr95570/ holds the 108-run battery (3 models, 3 reps,
2 arms) that validated the PR #95570 schema diet before merge:
base 49/54 vs diet 52/54, avg tokens/task -25%.
2026-08-26 08:24:21 -07:00
Teknium 21aa327c69 style: ruff lint + format on the eval scripts (encoding args, formatting) 2026-08-17 15:29:16 -07:00
Teknium 5033aedaa3 feat(evals): add the Browser Use mode A/B benchmark from PR #81958
Reconstructs the 204-run benchmark battery behind #81958 (Browser Use CLI
3.0 mode) as a rerunnable eval under evals/browser-use/, following the
toolperf_abeval / evals-compaction pattern.

- tasks/easy.json + tasks/hard.json: the oracle-checked toscrape task
  batteries (5 easy, 6 hard) exactly as run for the PR
- single_run.py: one cell = task x arm (base | pr | prns) x model x rep;
  throwaway HERMES_HOME, web-fetch creds stripped, arms pinned to separate
  trees via BUBENCH_BASE_TREE / BUBENCH_PR_TREE
- orchestrate.py: resume-safe local-CDP battery driver
- orchestrate_cloud.py: backend matrix (nous-cloud via the browser_use
  provider plugin, browserbase via REST) with per-cell session lifecycle
- report.py: scorecard aggregation with vs-base token deltas
- README.md: design, run instructions, and the recovered Aug 8-10 2026
  baseline scorecards (hard battery, backend matrix, easy round 1,
  digest ablation)

The original /tmp/bu-bench workspace was lost to a tmpfs reboot; harness
and readouts were recovered verbatim from the benchmark session's tool-call
history in state.db, with hardcoded paths parameterized. Smoke-verified
live: report.py aggregation, and single-cell runs (pr + base arms) against
a real headless Chrome CDP with sonnet-5 driving browser_exec, oracle pass.
2026-08-17 15:29:16 -07:00
Teknium 0ee5ae61e2 docs(evals): real Codex CLI head-to-head arm + results
scripts/codex_arm.py drives OpenAI Codex CLI end-to-end on the same
transcripts: chunk-file reads until its REAL auto-compaction fires (verified
via compacted events in the rollout jsonl; peak 455-483K vs its 258K
window), then quizzes post-compaction with the identical question banks and
judge. Results (results/codex-arm-2026-08-15/): codex 36.7% avg vs lean
closed-book 40.0% vs lean+recovery 68.3%. Codex has no runtime re-access
over its rollout history — the session_search differentiator, measured.
2026-08-15 18:41:24 -07:00
Teknium e2990428e7 docs(evals): ship transcript-building scripts + full eval detail in-repo
- scripts/reconstruct_lineage.py: rebuild full uncompacted lineage
  transcripts from a state.db COPY (descendant-tree walk, content-hash
  dedupe, synthetic-artifact strip, system_prompts hash resolution)
- scripts/replay_lineage.py + scripts/build_html_report.py: replay a 500K
  prefix through any checkout's compressor and render before/after
  side-by-side with compaction artifacts color-coded
- README: transcript-building workflow, scoping-tripwire section
- results/SCORECARD-2026-08-15.md: full per-transcript scorecards, all 4
  exam question banks, survival analysis, methodology + caveats
2026-08-15 17:40:43 -07:00
Teknium 9146f4c851 fix(evals): explicit encoding on all file I/O (ruff PLW1514) 2026-08-15 17:07:46 -07:00
Teknium 44536bd41b docs(evals): 4-transcript compaction scorecard
lean+recovery 68.3% avg recall @ 49K retained vs current 45.8% @ 162K —
+22.5pts at 0.30x tokens. Anchor index moved GUI needle-fact recall
23.3->60.0 closed-book, 46.7->80.0 with recovery.
2026-08-15 17:01:22 -07:00
Teknium c4bbb14e52 feat(compression): mechanical anchor index + region-scoping tripwire
- _build_anchor_index(): regex-harvests PR/issue numbers, SHAs, branches,
  file paths, error strings, handles, URLs from the compacted region into a
  bounded indexed summary section. LLM-free, so needle identifiers cannot be
  paraphrased away (the GUI-lineage failure class: 10/15 verbatim-or-nothing
  golds). Doubles as session_search query-anchor map.
- evals/compaction/test_region_scoping.py: sentinel tripwire proving the
  summarizer input carries ONLY the compacted region (head/tail sentinels
  never reach the serialized turns body) in both legacy and lean modes.
2026-08-15 17:01:22 -07:00
Teknium 7a82457ede feat(compression): digest noise filter + FTS5 recovery sim + digest-aware query hints
- _digest_worthy() drops no-signal tool rows before chunking (GUI-lineage
  digests were starving on tool-noise)
- eval recovery sim now uses in-memory SQLite FTS5 + BM25 (production
  session_search engine) instead of term-frequency scoring
- recovery query writer sees the digest section (front of context) so it can
  mine anchor identifiers
2026-08-15 17:01:22 -07:00
Teknium 8fe9025abd feat(compression): lean tail mode + recovery-aware eval arm
Lean mode (tail_mode='lean', default stays 'legacy'):
- tail budget = clamp(2.5% of window, 10K, 25K) instead of 0.20*window
- stale tail tool results demoted to session_search recovery stubs
- chunked identifier-preserving digests of the compacted region (map-reduce,
  pristine pre-prune tool contents)
- verbatim user messages embedded in summary (codex retention-by-role rule)
- deterministic session_search recovery footer

Eval: policies matrix gains lean + a '+recovery' arm giving the answerer one
simulated session_search round-trip against the archived region.
2026-08-15 17:01:22 -07:00
Teknium 33242d5ee0 feat(evals): compaction recall eval harness
Measures recall accuracy vs tokens retained across compaction policies.
Real transcripts in, LLM-generated recall exam from the summarized region,
per-policy answer+judge passes, scorecard out.
2026-08-15 17:01:21 -07:00
Teknium fd452e26e3 feat(tools): unicode-equivalent filename retry + near-miss suggestions in read_file
NFC/NFD, narrow no-break space (U+202F), and curly quotes render
identically in a terminal — a model retyping a visually-correct path
gets 'file not found' and can never discover the byte mismatch on its
own. On not-found, canonicalize the requested name and compare against
directory entries; exactly ONE equivalent spelling reads transparently
with an explanatory note. Zero or several matches (homoglyph twins)
fall through — never guess between collisions.

Also: difflib.SequenceMatcher >=0.8 fallback in _suggest_similar_files
catches near-miss typos (AGENT.md -> AGENTS.md) that substring scoring
misses entirely.

Measured (file-only arm, 3 reps, control=guard-only vs feature):
unicode task qwen3.8-max 31k->16k tok (-48%), turns 6.7->3.7;
opus-4.8 57k->33k tok (-42%), turns 8.3->5.0; accuracy held 1.00.
near-miss: opus mildly better, qwen flat, no regressions.
2026-08-09 23:35:07 -07:00
teknium1 ddd21abfa8 chore(evals): track results/.gitignore (its own * rule excluded it from the original add) 2026-08-09 23:30:02 -07:00
Teknium 0e63ed1feb feat(tools): stat-based special-file guard for read_file + readtool eval harness
read_file on a workspace FIFO/socket blocked until the exec timeout —
the existing device guard is name-based (/dev/*, /proc/*) and cannot
see an arbitrary special file. Add _special_file_kind(): one os.stat
on the resolved path, refusing FIFO/socket/char/block devices with a
plain note ('no read was attempted') instead of hanging. Host-visible
filesystems only; regular files, dirs, and missing paths unchanged.

Also adds evals/readtool/: an A/B harness that runs the real AIAgent
against hostile-file fixtures (huge lockfile, one-line bundle, FIFO,
NFD filenames, lying extensions) and measures accuracy, turns, tool
calls, and tokens. Measured for this guard (3 reps, file-only arm):
qwen3.8-max fifo task tokens 122k -> 26k (-79%), turns 9.3 -> 5.0;
opus-4.8 tokens 40k -> 23k; accuracy held 1.00 both arms.
2026-08-09 23:30:02 -07:00
teknium ba3fea24f1 Enhance TerminalBench 2 configuration and evaluation handling
- Added task_timeout parameter to enforce a maximum wall-clock time for each task, automatically scoring as FAIL if exceeded.
- Introduced terminal_timeout and tool_pool_size parameters to improve command execution and concurrency management.
- Updated logging to provide detailed task execution times and timeout handling, enhancing overall monitoring.
- Removed outdated evaluate_config.yaml file to streamline configuration management.
2026-02-10 22:53:24 +00:00
teknium 35ad3146a8 Add new environments and enhance tool context functionality
- Introduced new environments: Terminal Test Environment and SWE Environment, each with default configurations for testing and software engineering tasks.
- Added TerminalBench 2.0 evaluation environment with comprehensive setup for agentic LLMs, including task execution and verification.
- Enhanced ToolContext with methods for uploading and downloading files, ensuring binary-safe operations.
- Updated documentation across environments to reflect new features and usage instructions.
- Refactored existing environment configurations for consistency and clarity.
2026-02-10 19:39:05 +00:00