The browser vault now draws from three login sources behind one handle
shape: the local encrypted vault (vault_…), 1Password Login items (op:…)
and Bitwarden Password Manager logins (bw:…). browser_vault_list aggregates
metadata across them; browser_vault_fill routes by prefix and resolves the
password at fill time only, through the manager CLI.
External managers are locked until the user unlocks them for the current
session. The new browser_vault_unlock tool (and the fill path, implicitly)
asks the surface to show a masked master-password prompt — CLI panel
(reuses the sudo panel state), TUI/Desktop via a vault.unlock.request
blocking card. The password goes to `op signin --raw` / `bw unlock --raw`
on stdin, never argv or env; only the session token is kept, in memory,
with a 30-minute idle TTL, cleared on session close or `vault.lock`.
Headless contexts (cron, webhook, api_server, -q) can never prompt: the
manager is reported as locked with unlock=unavailable_in_this_session and
fill refuses — the same posture approvals take where nobody can answer.
Config: vault.onepassword / vault.bitwarden {enabled, binary_path, …};
a 1Password service-account token skips the prompt for headless use.
RPC: vault.sources, vault.source.set, vault.unlock, vault.lock for Settings.
Tests (2, real subprocess against a fake bw; each proven red by sabotage):
headless never prompts or spawns; unlock feeds stdin only, token never
enters os.environ, fill routes by prefix and the password only reaches the
fill script.
The facade forwarded turn_author= to the loop's public entry point, which did not
declare it: every real AIAgent.run_conversation() turn raised TypeError (16 CI
failures across provider, sidecar, cron and finite-chat suites). The PR's tests
only exercised build_turn_context directly, so the missing hop was invisible.
Adds one facade-through-loop test that goes red when the kwarg is dropped.
Keep one or two behaviour tests per seam (author reset on a cached agent, forged
_turn_author refused, guard trips and cools, single charge on the busy path) and
drop the parser/setting enumerations. a2a_key goes with them: nothing in this PR
reads it; the honcho follow-up that does can bring it back with its consumer.
message_agent sent a bare bot:<profile> id through hermes peer dm, so a remote coder and the recipient's own coder shared one author id. The peer branch now sends bot:<hostname>/<profile>, with the hostname cleaned like any author field and slashes dropped. The direct local path keeps the bare id.
A relayed DM stamped bot:<profile> on the recipient turn, so an ops profile on another machine and the local ops profile shared one author id. The Desktop now forwards from_connection with each bot_relay.deliver, and delivery_turn_author builds bot:<connection>/<profile> for it while the Desktop's own gateway ("local") keeps the bare id. An api author object accepts an optional origin string that yields the same shape.
MemoryManager.on_turn_start forwarded the author kwargs to every provider and _each_provider swallowed the TypeError, so a provider with the two-positional on_turn_start(n, text) stopped running. The kwargs are now filtered against the provider's signature the way _provider_sync_accepts filters sync_turn.
The Telegram adapter asks the authorization check before dispatch, the ingress gate asks it
again, and the busy path asks a third time. Each call counted one loop-guard event, so a
Telegram bot tripped the budget after a third of the configured messages. The verdict now only
refuses a chat that is cooling down. The ingress gate counts an admitted bot message once.
`parse_turn_author` treats only booleans, integers and the strings true/1/yes as a bot flag,
and returns None for an author with neither id nor name. Names keep format characters and
non-breaking spaces so emoji sequences survive. The quiet one-shot pops HERMES_TURN_AUTHOR
before the turn so tool subprocesses do not inherit it. `max_events` must be a whole positive
number. Issue numbers move out of code comments.
on_turn_start already received the author trio. sync_turn did not, so a
provider that wanted to write the turn under its author had to stash state
between the two hooks. sync_turn now takes turn_author as a keyword-only
argument, and MemoryManager sends it only to providers whose signature
accepts it, so existing providers keep working unchanged.
build_turn_context resets the author on the agent at the start of every turn
so a cached gateway agent never carries a bot author into the next human
turn. agent/turn_author.py holds the parsing and the HERMES_TURN_AUTHOR
carrier.
MemoryProvider.identity_signature() is a new optional hook: the identity
values a provider writes under, declared by the provider itself, for the
gateway's agent cache to key on.
`on_turn_start` documents a per-turn kwargs channel — "kwargs may include:
remaining_tokens, model, platform, tool_count" — and `MemoryManager`
forwards whatever it receives. Its only caller passed nothing, so a memory
provider had no way to learn who wrote the turn it was being told about.
Providers that key durable state on identity resolve one identity when the
session is created. A shared session does not work that way: threads are
shared by default (`thread_sessions_per_user` is False), so alice, bob, and
another agent all write turns into a session whose peer is whoever spoke
first. The gateway's answer today is the `[name]` prefix it prepends to the
message text, which the model reads and a provider cannot.
`turn_author` now travels from the gateway through `run_conversation` into
`build_turn_context`, which forwards `author_id`, `author_name`, and
`author_is_bot` to every provider. It stops there — the trio never reaches
the model, and providers that ignore the kwargs are unaffected.
The bot flag is sent on every transport, not only shared sessions: a
provider deciding whether a turn may write to durable memory needs it in a
DM too.
`SessionSource.is_bot` is only as good as its producers. `build_source`
defaults it to False and 3 of 32 adapter call sites pass it, so most
platforms still report every author as human. Populating the rest is
follow-up work; nothing here depends on the flag being right yet.
Three defects found in review of the first head (@ehz0ah):
* The mid-request hop read the PERSISTED provider from disk. A live
`/model xai-oauth` session over `model.provider: auto` therefore still
reached the discovery chain and billed Nous. _try_payment_fallback now
takes the route's main_runtime snapshot; disk is the fallback only when
no session runtime exists.
* After a configured fallback was quarantined mid-request (401, refresh
failed), the second pass went straight to the discovery chain, which
the new gate refuses — so later CONFIGURED entries never ran and the
original error was re-raised. The second pass now re-walks the task
chain and main chain (the quarantined entry is unhealthy and skipped)
before discovery.
* current_provider_owns_vendor dropped ids detect_vendor could not
classify, so Bedrock's 15-id catalog (14 unclassified `us.anthropic…`)
looked exclusively DeepSeek and `/model deepseek-v4-pro` stuck on
Bedrock. An unclassified id now counts as evidence of a multi-vendor
catalog: ownership requires every id to classify to the one vendor.
With a main provider selected, an unusable main route (expired xAI/Codex
OAuth token, 401/402/429 mid-session) fell through the built-in discovery
chain (OpenRouter -> Nous -> custom -> api-key) and quietly ran every
compression, title and memory-flush call on whichever OTHER account was
still logged in. Reported as "using Grok on my Premium+ sub, my Nous
Portal balance kept draining" — the chat visibly stayed on Grok while the
side tasks were billed elsewhere, and re-logging into X did not help
because the aux side never consulted the selected provider.
The discovery chain is now reserved for installs with no selected main
provider (`model.provider: auto` / unset). Otherwise the ladder is
main -> auxiliary.<task>.fallback_chain -> fallback_providers -> refuse
with a warning naming the dead provider and the fix. Both entry points
gate on the same predicate: the resolve-time route and the mid-request
payment/auth hop (_try_payment_fallback).
Existing chain tests that asserted the hop now pin `provider=auto`, the
one case where discovery is still the contract.
DeepSeek retired deepseek-v4-flash on 2026-09-10 (V4.1-Flash release); the API's
model name is now `deepseek-flash` and /v1/models lists only it. Hermes still
folded every non-V-series name onto deepseek-v4-flash, so `/model deepseek-flash`
on the DeepSeek provider was rewritten, then the validator "auto-corrected" it
back against the live listing: "Auto-corrected deepseek-v4-flash -> deepseek-flash"
on every switch.
Retired aliases (deepseek-chat / -reasoner and other fuzzy names) now fold onto
deepseek-flash; the curated catalog, profile fallback list, aux default, goal-judge
hint and pricing snapshot (2026-09-10 off-peak USD) follow the docs. Dated
deepseek-v4-* ids still pass through untouched.
Builds on YipTszkwan's #107126 (earliest fix in the cluster).
DeepSeek's 2026-09 Flash refresh introduced a version-less canonical id:
GET /v1/models now returns `deepseek-flash` (alongside `deepseek-v4-pro`), the
API accepts it directly, and the older `deepseek-v4-flash` is server-side
aliased onto it. Every DeepSeek model-id gate in Hermes keys off the
`deepseek-v<N>` prefix, so the new id silently missed all four:
* DeepSeekProfile.build_api_kwargs_extras classified it as non-thinking and
omitted `extra_body.thinking`. The server then defaults to thinking-on, so
the user's thinking toggle and `reasoning_effort` were quietly ignored.
* `_normalize_for_deepseek` folded it onto `deepseek-v4-flash` (it misses the
V-series regex), so the id a user picked never reached the wire and the
config stored a different model than the picker advertised.
* `DEFAULT_CONTEXT_LENGTHS` fell through to the 128K `deepseek` catch-all
instead of the real 1M window, capping the model at an eighth of its
context before compaction kicked in.
* `_REASONING_STALE_TIMEOUT_FLOORS` had no entry, leaving the stale-stream
detector at its 180s default instead of the 600s reasoning-model floor.
Verified live against api.deepseek.com: `deepseek-flash` answers 200 with
`model: deepseek-flash`, accepts image input (the refresh folds vision into
the Flash model), and the in-between id `deepseek-v4.1-flash` is rejected
with "The supported API model names are deepseek-flash, deepseek-v4-pro".
Adds the id to all four gates plus regression coverage for each site.
Every GLM slug that missed a DEFAULT_CONTEXT_LENGTHS key fell to the "glm" 202,752 catch-all, so
compression fired at ~20% of the real window and the aux feasibility check auto-lowered the session
threshold to that number (the 202,752 in the Coatue compression report).
- Catalog: GLM entries from Nous + OpenRouter /v1/models (2026-09-09): 5.3 / 5.3-flash 1,310,720
(:batch/:US 1,048,576), 5.2 1,048,576, 5 / 5.1 / 4.7 / 4.6 204,800, *-turbo / 4.7-flash 202,752.
- _catalog_key_matches: version separators normalised on both sides so relay slugs like z-ai-glm-5-3
hit glm-5.3 (#97398; approach from PR #97412 by @shellybotmoyer, re-implemented on current main).
- get_custom_provider_context_length: entry-level custom_providers[].context_length backs every model the
entry serves when no per-model override exists, so /model switch stops dropping it (#98387; from
PR #98396 by @liuhao1024, re-implemented on the config_providers sibling).
- check_compression_model_feasibility: when aux compression is the main model on the main route, reuse the
main model's resolved window instead of re-resolving without its pin (#89500 mechanism 2, #45519).
Closes#97398, #98387, #89500, #87825, #97595, #97820 (suffix strip already on main; catalog values now match).
A rejected OAuth token (HTTP 401 'User not found' from Nous Portal, Codex,
xAI…) reached the desktop error card as 'Provider error' with Retry as the
first action, which just replays the same dead credential.
Backend: nonretryable_client_error_result dropped failure_reason /
failure_retryable, so error_surface classified every non-retryable 4xx as a
retryable provider failure. It now stamps the classifier verdict like the
max-retries path, and auth-layer descriptors carry auth_kind
(oauth|api_key, derived from the provider catalog tab) + provider_label.
Desktop: an auth/oauth surface renders 'Authentication error', explains that
the <provider> sign-in expired/was revoked, and offers 'Sign in to <provider>
again' which launches that provider's existing onboarding OAuth flow scoped
to the failed session's gateway profile. Retry stays as the follow-up click.
Re-login to the provider already in use keeps the current model instead of
swapping in the recommended default.
Follow-up to #106866 from the post-merge audits (@ehz0ah, @victor-kyriazakos):
- _recoverable_pool_provider: the "endpoint mismatch, not a dead key" shield now (a) compares the full
origin via base_url_origin() (scheme+host+port — a different port or an HTTPS->HTTP downgrade is a
different trust boundary) and (b) applies only when the rejected client carried the SESSION's key, so
an independently owned auxiliary pool keeps rotating at its own configured origin.
- _resolve_named_custom_branch: explicit/per-task base_url and api_key compose OVER the providers.<name>
entry field-by-field instead of the entry silently replacing them (URL-only, key-only, both, neither).
- _acreate_with_progress: plain-create fallback only for a rejected stream NEGOTIATION (mirrors the sync
wrapper); a mid-stream failure propagates to the classified recovery ladder instead of re-sending the
whole prompt non-streaming.
Tests (all red on main): foreign origin x {host, port, scheme} shields the session key; same origin and
independent aux pool still rotate; named-provider override matrix through resolve_provider_client on a
real temp HERMES_HOME; async content-then-error is not retried non-streaming.
`auxiliary.<task>.provider: openai` was rewritten to custom + api.openai.com/v1 unconditionally, and the
api-key discovery rung used the registry default endpoint, so proxy/gateway users (OPENAI_BASE_URL or
providers.openai) had compression hop to the public endpoint with a proxy-issued key -> 401 -> the
key/pool was quarantined and the session sat over the compression threshold.
- _expand_direct_api_alias: a providers.openai entry keeps its name (named-custom branch applies its
base_url/key); otherwise OPENAI_BASE_URL wins over the public default.
- _resolve_api_key_provider: when the bound session runtime is that provider, use its endpoint + key.
- _recoverable_pool_provider: a rejection at a host other than the session's configured endpoint for
the same provider is an endpoint mismatch, not a dead key -> no rotation / unhealthy mark on the pool.
Complete the seam default from the salvaged commit: _relay_async_completion needs an async twin of
_create_with_progress, and the async primary attempt previously streamed only for stream-only
providers, so a hooked async compression call ticked the watchdog zero times.
Adds one invariant test: with a hook installed, BOTH relay defaults stream and tick per chunk;
without a hook they are byte-identical plain creates. Red on origin/main.
_is_azure_foundry_responses matches the azure-foundry provider or the
project-scoped services.ai.azure.com host. #101243 proposes narrowing it
to that host alone. The multi-item rejection this branch handles happens
on resource-level *.openai.azure.com hosts too, so the pruning gate now
uses _is_azure_responses: the provider, or either Azure host.
Refs #105369
The first request of every turn after the first replays the encrypted
reasoning item of every earlier response. Azure Foundry accepts one such
item and rejects two or more with HTTP 400 "Conflicting authenticated
continuation identities". In-turn requests never hit this because
_is_post_tool_replay already suppresses replay on Azure when the request
ends on a tool result.
On Azure Foundry, build_kwargs now keeps codex_reasoning_items only on the
newest assistant row that has any. include still requests
reasoning.encrypted_content, compaction checkpoints stay on every row, and
the canonical messages are not mutated. Other Responses endpoints are
unchanged.
Rebuilding a failed session from state.db through the transport and
sending it live: unpatched, 5 reasoning items, 400; patched, 1 item, 200.
Fixes#105369
Azure Foundry (gpt-6-astra) answers a request that replays encrypted
reasoning from more than one prior response with HTTP 400 "Conflicting
authenticated continuation identities", code invalid_value. The classifier
mapped that to a non-retryable client error, so the turn aborted and every
later turn in the session failed the same way.
Map the message to invalid_encrypted_content. The existing recovery in
turn_recovery then strips the replay state and retries once.
Refs #105369
finish_reason='length' has two causes: the answer was long (max_tokens reached),
or the prompt itself left no room to generate. _continue_text treated both the
same: append the fragment + a continuation nudge and retry, up to 4 times. In the
second case every retry sends a strictly longer prompt, so each attempt is worse
(Ollama n_ctx=32768: 32,638 -> 32,685 -> 32,732 prompt tokens, all truncated), the
user is told "model hit max output tokens", and max_tokens is not the lever.
The response's usage already carries prompt_tokens and the compressor already
resolves the model's context window; compare them once per truncation. Under
_MIN_CONTINUATION_HEADROOM (512) free tokens the turn ends on the first
truncation, keeps the partial text, names the context window as the cause and
points at /compress or a larger window. Unknown usage or window keeps today's
behaviour. max_tokens semantics untouched.
Co-authored-by: gaoanze888 <214786078+gaoanze888@users.noreply.github.com>
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Novita returns HTTP 429 with message 'server overload, please try again
later' and error type 'server_overload' when its server is genuinely busy.
Neither phrase was in _OVERLOADED_PATTERNS, so the 429 fell through to
_V_RATE_LIMIT and set should_fallback=True + should_rotate_credential=True —
rotating the credential / falling back early instead of retrying the same
key. Add 'server overload' and 'server_overload' to the overload tuple so
this reaches the existing _V_OVERLOADED verdict (retryable, no rotation).
Closes#106205
Collapse the four class-based tests into two parametrized invariants over both
refunding restart flags and move them to tests/agent/ (the phase modules live in
agent/): a single restart still refunds-and-continues; a re-armed restart breaks
after max_retries refunds. The stub grows the redirect seam the follow-up commit
uses so the queued-correction contract is covered by the same test.
Drop the sentinel-only batch test: a batch sentinel is already rejected by the
shared _is_clarify_non_response_sentinel list check that the existing sentinel
tests pin, so the case adds no new contract. Also add the contributor email
mapping for the cherry-picked commit so release CI can attribute it.
_sum_clarify only extracted the top-level ``user_response`` key, so batch clarify
results (questions=[...] -> responses[].user_response) fell through to the generic
placeholder and the summarizer never saw the user's answer/permission decision.
Closes#106077.
A fresh process with an unchanged store takes zero auth-store locks, and a
root store that gains a forked grant invalidates the persisted mark so the
heal re-runs. The corrupt-mark fallback, same-mtime size change and
no-secrets checks were pinning implementation details of the same cache.
`_heal_forked_single_use_oauth_grants()` runs on every `load_pool()`, and its
clean mark lived only in memory. Every fresh `hermes` invocation and every new
worker therefore re-took the heal's two nested EXCLUSIVE auth-store locks just
to rediscover a store it had already cleared — 4 acquisitions per process on a
two-provider profile, every one of them finding nothing to consolidate. Behind
a sibling process holding those locks that costs a full
`AUTH_LOCK_TIMEOUT_SECONDS` per provider before the process can do anything at
all: measured 30.1s for two providers.
Persist the mark next to the store it describes (`<profile>/cache/
oauth_heal_clean.json`, 0600, paths and stat data only — no credential
material) and consult it BEFORE taking any lock.
Outliving the process means the mark needs a stronger key than the in-memory
one did:
- The ROOT store joins the fingerprint. This heal consolidates root → profile,
so root acquiring a counterpart turns a row the heal deliberately KEPT into a
fork it must strip. The in-memory mark could ignore root because it died with
the process; a persisted mark would keep skipping a heal that has become
necessary.
- File sizes join it too, so a metadata-preserving rewrite (`rsync -t`,
`tar -p`, a restore) cannot leave a stale mark looking current indefinitely
rather than for one process.
Measured on an isolated HERMES_HOME with two OAuth providers:
lock acquisitions per fresh process 4 -> 0
contended load_pool() x2 30.1s -> 0.00s
mark-file reads per 100 load_pool() - -> 2 (one per provider)
The mark stays a cache: absent, unreadable, corrupt or wrong-shaped content all
mean "unknown" and fall through to the locked heal, and a failed write only
means the next process re-runs it — the behaviour before this cache existed.
_maybe_inject_run_budget_wrapup() appends its wrap-up notice to the newest
role:"tool" message in place, with no _DB_PERSISTED_MARKER check. Its sibling,
_maybe_inject_iteration_budget_warning(), got exactly this guard added in the
same recent saga (turn_iteration_prep.py), with the comment "an older turn may
already be cached."
The reachability is structural, not an edge case: _maybe_inject_run_budget_wrapup
is only ever called from prepare_iteration(), at the START of the next iteration
-- strictly after tool_executor.py's _flush_session_db_after_tool_progress has
already flushed and marked the previous iteration's tool row persisted. So every
successful injection was mutating an already-persisted row: the wire request for
that turn carried the notice, but the durable transcript never did, diverging
replay from the live bytes and invalidating the provider's prompt-cache prefix
from that row onward.
Fix:
- Add the same _DB_PERSISTED_MARKER guard to _maybe_inject_run_budget_wrapup,
scoped to the specific tool row the reversed scan lands on (not just
messages[-1], since this function -- unlike its sibling -- scans backward for
the newest tool row rather than only checking the tail).
- Wire _maybe_inject_run_budget_wrapup into _flush_session_db_after_tool_progress
(pre-flush), mirroring exactly how _maybe_inject_iteration_budget_warning is
wired in both places. Without this, the guard alone would make the notice stop
firing in the common case, since prepare_iteration's call site almost always
hits an already-persisted row -- the pre-flush call site is what actually lets
it land in durable bytes.
Verified empirically: read the real call graph (tool_executor.py's three
_flush_session_db_after_tool_progress call sites cover every tool-completion
path) to confirm the guard's premise, then added an end-to-end test using a real
AIAgent + SessionDB that flushes and checks the persisted row for the notice
text. Mutation-verified: reverting the two production files drops exactly the 2
new/updated assertions (28 pass, 2 fail); reapplying restores green (30 passed).
Also ran the sibling iteration-budget-warning and /steer suites (71 passed) to
check for interaction regressions -- none.
The ledger in verification_evidence.db exists only to feed the verify-on-stop
guard, but the recorder kept running on every foreground terminal command and
every file edit after #53552 turned the guard off by default. Users who never
opted in still accumulated a multi-MB database (7 MB / 4.6k rows on one install).
Every ledger entry point (record_terminal_result, record_verify_run,
mark_workspace_edited, verification_status) now checks verify_on_stop_enabled()
first and returns without opening or creating the database when the guard is
off. verification_status reports {"status": "disabled"} in that case; no client
consumes the verification.status RPC yet, so nothing downstream changes.
Existing ledger tests pin HERMES_VERIFY_ON_STOP=1 since they exercise the ledger
itself; the new test proves the off path never creates the file (red on base).
With account-identity matching gone, the providers.<id> block consolidation
only fired on shared token material. A historical fork (same copied pool-row
id, profile rotated, both pairs diverged) then healed the pool row into root
but left root's providers.openai-codex block on the spent pair; root's next
load_pool() re-seeds its device_code row FROM that block and undid the heal.
_HealPass now records that a profile pool row matched root by copied id or
shared tokens and passes that verdict to _heal_forked_provider_block, which
accepts it as lineage proof. No account-identity guessing is restored; an
independent same-account grant (no id/token match) is still left alone.
Follow-up to simpolism's #106177.
`kwargs.get("target") is sched._run_callback` is always False (a bound method is a fresh object
per access), so the fake never returned Boom and the _dispatch failure branch went untested;
the test passed on the normal worker. Compare with == and assert the interception happened
(mutation: retiring the handle on start failure now fails the test).
The thread-count assertions sampled while per-fire workers were still live; quiesce every
handle with cancel(wait=) before sampling so the count is deterministic (AGENTS.md: timing tests
must not assume a quiet runner).
Follow-up to #106308.
The salvaged commit forked an explicit /refine under a new "refine_review" origin so
the memory delete gate would not treat it as unattended. But is_background_review()
is the key for every other review guard — skill_manager_guards (curator-owned-only,
read-before-write), skill_manager_tool (archive instead of rmtree), skill_ledger
actor, write_approval staging, the [auto] tag — so a /refine fork silently escaped
all of them.
Carry attendedness separately: the fork keeps origin "background_review" and sets
_review_attended; turn_context binds it beside the origin ContextVar; the memory
gate keys on the new is_unattended_review(). Also run the gate AFTER
_validate_single_op / the operations list check, as memory_tool's own docstring
requires, so an invalid replace is rejected now rather than staged and failed at
approve time.
Review follow-up on #105944 (#105921):
- explicit /refine forks now run under the refine_review write origin
(explicit flows from the CLI/gateway handlers through
_spawn_background_review_now and spawn_background_review_thread down
to build_cache_parity_fork), so a user-requested review keeps the
full memory operation set; only automatic reviews stay behind the
unattended delete gate.
- the unattended delete gate now stages the denied replace/remove (or
whole batch) into the pending store instead of dropping it: the
fork's own review summary is never published, so a plain denial lost
the consolidation request with no surfacing path. The staged proposal
carries a proposal_staged marker that summarize surfaces as an action
line, and a staging failure still fails closed to a plain denial.
- regression tests: explicit-path origin pass-through, refine_review
keeping replace working, near-limit denial end to end (add rejected
by budget -> replace staged -> proposal surfaces, store unchanged).
The review fork's tool whitelist granted the whole memory toolset
whenever the profile had memory enabled, regardless of which nudge
fired, so a skill-nudge fork held remove/replace on MEMORY.md it was
never asked to use; combined with the memory tool's near-limit
'consolidate now' hint, an unattended fork deleted standing rules with
no user in the loop.
- Pass review_memory from spawn_background_review_thread through
_run_review_in_thread/_run_review_fork into _review_tool_whitelist;
a skill-only review no longer gets the memory tool at all.
- Fail-closed operation gate in memory_tool: a background-review fork
may add, never replace/remove (single or in a batch) — consolidation
decisions reach a human via the review summary instead.
- Keep the deny/prompt wording in sync with the whitelist so a
memory-less review doesn't advertise memory.
Fold the identical heappush(...) into PeriodicScheduler._requeue; notify() instead of
notify_all() now that the scheduler thread is the only condition waiter; drop the
PR-history paragraph from the module docstring (the commit carries it).
Tests: the blocked-sibling test asserted `sibling_ran.wait(0.30)` — a wall-clock bound
under the repo's ≥2 s flake floor; it now asserts the sibling fired while the blocker
still held its worker. The worker-start-failure fake keys on this scheduler's own
_run_callback rather than the global thread-name prefix so a leaked handle on _DEFAULT
cannot consume the single injected failure. The base-green no-overlap test is dropped
(it does not prove the fix).
oMLX/MLX prefill memory-guard rejections name an allocation peak in BYTES but
close with "Reduce context length", so _CONTEXT_OVERFLOW_PATTERNS claims them
and the turn enters the compress-and-shrink loop. Compression cannot lower a
prefill peak — the prompt is usually far below the window — so it burns the
compression budget, re-hits the wedged server on every attempt and ends in
"Cannot compress further" plus a destructive session reset.
Classify them as `overloaded` instead: retry with backoff, no compression, no
session reset (mirrors 503/529 recovery).
The guard runs before the overflow check AND before the usage-limit
disambiguation. The second ordering matters more than it looks: "memory limit
exceeded" contains "limit exceeded", so a status-less rejection — a proxy that
flattened the body — is currently classified as `billing` and rotates a
healthy credential.
Sites covered:
- _OVERFLOW_AS_5XX_RULES, which _400_TAIL_RULES extends → 400, 500, 502,
503, 529
- _MESSAGE_HEAD_RULES for the status-less path (ahead of usage-limit)
- _ERROR_CODE_VERDICTS for the structured oMLX codes
- _classify_400, because _by_status runs before _by_error_code, so a body
whose wording a proxy stripped would otherwise fall through to
format_error
Every pattern names memory/allocation in bytes, never a token or window count,
so the list stays disjoint from _CONTEXT_OVERFLOW_PATTERNS. Both oMLX wordings
are kept: 0.5.6 says "Prefill would require ~13.87 GB peak", 0.5.7 reworded it
to "predicted peak would require/exceed" and both are in the field. A test
pins that a genuine window overflow still compresses.
Refs #52261. Supersedes #52289, which predates the classifier rewrite and can
no longer be merged.
_session_db is always a SessionDB (agent_init / delegate_tool), so the
"fail closed on a store wrapper" hasattr was defense around code that
cannot fail; the store's own guard binds the value into SQL, so the
prologue only needs the sibling idiom isinstance(_row_id, int) that
session_persistence and transcript_repair already use. The positional
hazard is explained once, on set_latest_user_api_content. The in-place
compaction test duplicated test_api_content_sidecar's
test_inplace_compaction_backfills_sidecar_into_db verbatim (its row_id
parameter was never varied); dropped, as was the positional-helper tail
of test_older_identical_row_is_untouched already covered there.
The turn-start stamp had grown its own copy of the "what does the current
user row hold" rule (persist override = clean transcript, live content =
wire bytes = sidecar when they differ) that _db_flush_row already
implements. Two copies drift; extract durable_user_row_content() in
session_persistence and call it from both.
Also: reuse _persist_lock() instead of a third open-coded lock/nullcontext
ladder; drop the hasattr guard on set_latest_user_api_content (it predates
this fix and exists on every SessionDB); cut the comment to the WHY;
trim the new test file from 18 cases to the 7 invariants (real close
flush E2E, repeated-"ok" positional protection, API-only pre-flushed
turn, normal path writes nothing, compaction keeps positional, store
guards). Still 3 red / 4 green when agent/turn_context.py is swapped
for main's copy.
The api_content sidecar ('persist what you send') preserves prompt-cache
stability across turn boundaries by persisting the exact API-bound bytes
(including memory-manager prefetch, plugin injections, and API-only notes)
and substituting them on replay.
When a user turn was already materialized in the database before the
sidecar could be composed (in-place preflight compaction or a close/early
flush racing the prologue on the CLI path), the turn-start crash persist
marker-skips that message. Previously, the backfill was gated strictly on
in-place compaction (_preflight_compressed and _last_compaction_in_place),
so racing CLI flushes left api_content = NULL in SQLite and broke prompt
caching on subsequent turns (#102194).
Positional approaches (such as #102239 and #102286) using LIMIT 1 on the
newest active user row are unsafe: repeated common inputs ('ok', 'yes',
'continue') cause the backfill to match and overwrite the PREVIOUS turn's
row with the new turn's sidecar, corrupting history and breaking cache parity.
Resolve all landing blockers and review feedback from #102411:
1. Bounded state owner (Sahilvishnaliya):
Add SessionDB.set_message_api_content(session_id, row_id, content, api_content)
to SessionMessagesMixin in hermes_state_messages.py instead of growing
hermes_state.py. Update set_latest_user_api_content docstring with durable
warning on the positional hazard.
2. API-only turns & durable content selection (ehz0ah):
When a pre-flushed clean input has an API-only difference (e.g. voice
prefix or model-switch note):
- Retain the differing API-facing bytes as api_content even when no
new memory or plugin context was injected.
- Derive the durable content guard using _override_replaces_content so
the SQL 'content IS ?' guard matches the clean override text stored
in the DB row rather than the restored wire text.
3. Turn prologue gating (_row_id) & fail-closed store duck-typing (ehz0ah):
In agent/turn_context.py::_stamp_api_content_sidecar: check _row_id on
the live user dict (stamped by _insert_message_rows and synced by
sync_flushed_message_markers). If valid (positive int, not bool), address
by exact ID. Do NOT fall back to positional matching when a row ID is
present: if an external or custom wrapper lacks set_message_api_content,
fail closed and skip rather than corrupting a neighbouring row. If absent
but in-place compacted, fall back to positional update. On normal turns,
skip the backfill entirely (single atomic INSERT).
4. Real lifecycle test coverage (salch-cred, ehz0ah):
Comprehensive tests in tests/agent/test_api_content_row_addressed_backfill.py
covering store guards, surrogate scrubbing, gate non-arming, older identical
row protection, real close-flush row_id synchronization, API-only clean
override preservation with exact wire replay, and duck-typed store fail-closed
verification when set_message_api_content is absent.
Fixes#102194.
Closes#102411.