Files
hermes-agent/agent
Teknium 22c5048d9c fix(moa): restore prompt caching for the aggregator and advisors (#57675)
Two caching holes made MoA re-bill essentially its entire input stream:

1. AGGREGATOR: anthropic_prompt_cache_policy() judged the agent's own
   model/provider — on the MoA path those are the virtual preset name and
   'moa', which match no caching branch, so _use_prompt_caching was False
   and the acting aggregator (Claude on OpenRouter) ran with ZERO
   cache_control breakpoints. Measured on identical opus-4.8 sessions:
   85% cache share solo vs 2% via MoA — ~30M re-billed input tokens on one
   132-task benchmark run. Fix: when provider == 'moa', resolve the policy
   from the preset's real aggregator slot (provider/model/base_url/api_mode
   via resolve_runtime_provider).

2. ADVISORS: _run_reference never applied cache_control at all, and
   Anthropic caching is opt-in per request — Claude advisors served 0
   cache reads across 1,227 benchmark calls (11.5M re-billed input tokens)
   even though the advisory view is append-only across iterations (stable
   prefix; the synthetic end marker is last so it never pollutes it). Fix:
   _maybe_apply_advisor_cache_control() reuses the SAME policy function and
   SAME system_and_3 layout as the main loop, judged on the advisor slot's
   own resolved runtime — advisor requests are now decorated exactly like
   an acting agent on that provider. Auto-caching routes (OpenAI-family)
   are left untouched by policy.

Live-verified on the wire (per-iteration opus+gpt5.5 preset, 4 fan-outs):
claude advisor fan-out 2-3 cache_write=2161/2344, fan-out 4
cache_read=2206 / fresh_in=2; aggregator session cache share 84%/77%
(vs 2%/0% before). Sub-1024-token prompts correctly stay uncached
(Anthropic minimum).
2026-07-03 04:08:48 -07:00
..