Teknium
a8ca904922
evals/postmortem/ turns the one-off audit behind tracking issue #103563 into something anyone with a Hermes state.db copy (and optionally rotated agent.log*) can run on their own fan-out: forensics/ common.py discovers the run tree (root = most descendants, compression-rollover children excluded so cost buckets stay disjoint), fits pricing from estimated_cost_usd, and five lanes recompute the OBSERVED figures: tokens (buckets, depth/duration shares, context reconstruction, excess-cache-write proxy, cap replay), logcalls (per-call cache behaviour from agent.log with coverage printed first; strict and loose plateau definitions reported separately), delegation (timeouts, orphaned children, polling hours, batch-join withheld child-hours, truncated summaries), tools (hardline blocks, foreground refusals, whole-file rewrites), goal_loop (nudges, parked barrier), rework (public-surface drop at PR open + post-open commit inventory). Every figure is labeled OBSERVED or MODELED. live_ab/ the per-PR A/Bs (real code paths, fake providers, temp HERMES_HOME), paths from argv. review_probes/ the independent /review's probes, credited and adapted; each reproduced a round-1 defect and the fixed head must pass it. run.py runs the offline probes against one or two checkouts and prints PASS/FAIL side by side (--live adds the ones that spend cents). tests/ synthetic-DB smoke test for the lanes and runner. On the run's DB the lanes reproduce the tracking issue's population exactly (1,394 sessions, 93,284 calls, $19,302.59; cache_write $11,159.76) and on main vs an integration checkout of the 13 PRs the runner shows every probe FAIL -> PASS (two guard-only probes pass on both, noted in run.py). The trajectories are deliberately not shipped: the DB holds 51,956 home paths, 5,341 e-mails, private IPs, chat ids and real-shaped credentials in tool output. The lane reports and recomputed JSON are in a secret gist linked from #103563.
57 lines
2.4 KiB
Python
57 lines
2.4 KiB
Python
"""#103507: pytest plugin forcing the consumer-first schedule (-p finalizer_schedule_probe --finalizer-probe=consumer-first)
|
|
|
|
Independent-review probe (written by the /review subagent for tracking issue #103563, adapted here).
|
|
It reproduced a defect in the first version of the PR; the fixed head must pass it. Paths are taken
|
|
from the command line / environment, never hard-coded. Usage: see the argument parsing at the top of the file.
|
|
"""
|
|
import asyncio
|
|
import json
|
|
import pytest
|
|
|
|
|
|
def pytest_addoption(parser):
|
|
parser.addoption('--finalizer-probe', default='off')
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def finalizer_schedule_probe(request, monkeypatch):
|
|
mode = request.config.getoption('--finalizer-probe')
|
|
if mode == 'off':
|
|
yield
|
|
return
|
|
from agent import relay_llm, chat_completion_helpers
|
|
print('PROBE_IMPORT', relay_llm.__file__, chat_completion_helpers.__file__)
|
|
original_provider = relay_llm.ManagedLlmStream._provider_stream
|
|
original_count = chat_completion_helpers._StreamingCall._count_chunk
|
|
stats = {'gated_terminal_chunks': 0, 'consumer_releases': 0}
|
|
|
|
def terminal(chunk):
|
|
choices = chunk.get('choices') or []
|
|
return (not choices and chunk.get('usage') is not None) or (
|
|
bool(choices) and choices[0].get('finish_reason') == 'tool_calls')
|
|
|
|
async def gated_provider(stream, *args):
|
|
async for chunk in original_provider(stream, *args):
|
|
if terminal(chunk):
|
|
stream._review_gate = asyncio.Event()
|
|
stats['gated_terminal_chunks'] += 1
|
|
yield chunk
|
|
await asyncio.wait_for(stream._review_gate.wait(), timeout=10)
|
|
else:
|
|
yield chunk
|
|
|
|
def count_and_release(call, diag, chunk):
|
|
stream = call.managed_stream_holder.get('stream')
|
|
gate = getattr(stream, '_review_gate', None)
|
|
raw = chunk.model_dump() if hasattr(chunk, 'model_dump') else vars(chunk)
|
|
if terminal(raw) and gate is not None and not gate.is_set():
|
|
gate.set()
|
|
stats['consumer_releases'] += 1
|
|
return original_count(call, diag, chunk)
|
|
|
|
assert mode == 'consumer-first'
|
|
monkeypatch.setattr(relay_llm.ManagedLlmStream, '_provider_stream', gated_provider)
|
|
monkeypatch.setattr(chat_completion_helpers._StreamingCall, '_count_chunk', count_and_release)
|
|
yield
|
|
print('PROBE_STATS', json.dumps(stats, sort_keys=True))
|