When a prompt_toolkit run_in_terminal cooked->raw restore is lost (cancelled
coroutine, racing chained cross-thread windows from background-review
summaries / process-notification prints), the tty stays in cooked mode while
the Application still expects raw. The kernel line-buffers keystrokes and the
CLI appears to stop taking input even though the event loop is healthy.
Observed live 2026-08-13: interactive session left in 'icanon echo' after a
background skill-review fork + notify_on_complete turn; only an external
stty rescue restored input.
Fix: _heal_cooked_mode_drift() re-applies prompt_toolkit's own raw-mode flag
surgery when stdin's lflag has drifted cooked, and process_loop's idle branch
runs a rate-limited _check_termios_drift() watchdog that skips legitimate
cooked windows (app._running_in_terminal), agent-running phases, non-tty
stdin, and Windows.
patch_session_model_config merges key-level and only deletes on explicit
None. Dropping falsy values from the top-level patch let a previous
switch's api_mode/base_url survive the next switch — TUI/desktop resume
then restored e.g. openrouter with anthropic_messages wire mode, and a
failed bare-custom heal produced a stale-provider/new-endpoint route.
Write absent top-level values as explicit None so each switch fully
replaces the persisted route. Regression test against a real SessionDB;
mutation-checked. Also correct the heal comment (CLI is deliberately
stricter than the TUI recovery, which keeps bare custom with a base_url).
- Bare 'custom' from ModelSwitchResult.target_provider is the resolved
billing class, not a routable identity; persisting it verbatim made a
later --resume hard-fail once the config default moved off the custom
endpoint. Heal to custom:<name> via canonical_custom_identity at
persist time, and again on restore for rows written by older builds
(mirrors tui_gateway's _stored_session_runtime_overrides recovery).
- --global switches now also update the session row: the row records
what THIS session runs, otherwise resume restored the stale
creation-time model over the user's new global choice.
- Only adopt resolved credential_pool alongside its api_key (don't null
the ambient pool when resolution returns no credentials).
- 3 new tests; healing path mutation-checked.
- Extract the two duplicated /model session-persist blocks into
_persist_model_switch_to_session; persist the route BOTH nested
(gateway_runtime, CLI reader) and top-level (TUI gateway's
_stored_session_runtime_overrides reader) so a CLI switch also
survives a desktop/TUI session.resume.
- Add SessionDB.session_gateway_runtime as the canonical tolerant
row-level route reader (session_yolo_enabled precedent); use it in
_restore_session_model instead of hand-rolled JSON parsing.
- Clear stale launch-time _explicit_api_key/_explicit_base_url when
resume restores a different provider (same leak guard
_apply_model_switch_result already has).
- 12 new tests incl. a real-SessionDB round trip; mutation-checked.
Two bugs caused resumed sessions to use the config default model instead
of the model the session was actually using:
1. CLI /model switch didn't persist the new model to the session DB row.
The gateway calls update_session_model() after a /model switch, but
the CLI path only updated in-memory state and the agent's runtime —
it never wrote the new model to the sessions.model column. So the DB
row always kept the original model from session creation.
2. Resume didn't restore model/provider from the session DB row.
_preload_resumed_session and _init_agent restored CWD and YOLO from
session_meta, but never read session_meta['model'] back into
self.model/self.provider. So even if the DB had the right model,
resume would use whatever was in config.yaml.
Fix:
- _handle_model_switch / _apply_model_switch_result: call
update_session_model() after a session-scoped /model switch (skipped
for --once and --global), mirroring the gateway's behavior.
- New _restore_session_model() method: restores model/provider from
session_meta on resume, with provider/base_url/api_mode from
model_config.gateway_runtime. Also swaps the running agent in-place
for mid-chat /resume.
- Call _restore_session_model() from all three resume paths:
_preload_resumed_session, _init_agent, and _handle_resume_command.
- Track _explicit_model_override flag so -m/--model on the CLI overrides
resume (user intent wins). Cleared on /new.
Follow-ups on salvaged #85287:
- discover_entrypoint_manifests() now carries BOTH the import-free kind
classification (from #85527) and capability declarations — the two
contracts compose in one function instead of the capability rewrite
dropping classification.
- Per-entry exception isolation: one malformed distribution no longer
blanks every other plugin's manifest (same contract as
providers/__init__.py entry-point scan).
- Documented the hermes_agent.plugin_capabilities group in the plugin
developer guide (pyproject example).
- Single-source the included note as _INCLUDED_NOTE and attach it at
BOTH status='included' sites (the zero-amount pricing-entry branch
previously returned the same status with no note).
- Docstring/comment precision on format_cost_label: the fallback
triggers on 4dp ROUNDING to 0.0000 (banker's rounding includes the
exact $0.00005 boundary), not truncation; note why the rendered-label
guard beats a naive Decimal threshold.
- Tests: replaced a dead assertion with the exact-boundary case
($0.00005), fixed an overclaiming comment, aligned the terminal
cost column.
- Insights formatters now route aggregate estimated cost through the
shared format_cost_label() instead of hardcoded 2dp — a sub-cent
aggregate (one cheap DeepSeek session, ~$0.0046) no longer renders
'Estimated: ~$0.00', the exact bug class this PR fixes (#79220).
- format_cost_label: positive amounts below $0.00005 render '~$<0.0001'
instead of the zero-looking '~$0.0000' 4dp truncation artifact.
- Renamed _format_cost_label -> format_cost_label (now a cross-module
shared helper).
- Tests: renamed test_gateway_format_hides_cost ->
test_gateway_format_hides_cache_details and
test_no_cost_section_when_all_zero ->
test_unknown_bucket_shown_for_costless_session (names contradicted
behavior); restored a real assertion in the custom-models test that
had been weakened to a comment; added sub-cent-aggregate and 4dp-floor
contract tests (mutation-checked).
Three cost-display honesty fixes:
1. Sub-cent cost label rendering (#79220) — _format_cost_label() scales
precision to magnitude: zero renders as '$0.00', sub-cent (< $0.01)
renders at 4 decimal places (e.g. '~$0.0046'), normal costs keep 2dp.
This fixes the bug where DeepSeek per-turn costs of $0.004640 rendered
as '~$0.00' despite amount_usd carrying full Decimal precision.
2. Cost bucket surfacing (#77223) — insights format_terminal and
format_gateway now display three cost buckets: estimated (with dollar
figure), included (session count, labeled 'subscription — no provider
invoice'), and unknown (session count, labeled 'no pricing data').
Previously, included and unknown sessions silently collapsed to $0 in
the aggregate view, hiding 315 of 473 sessions in the reporter's DB.
3. Subscription-included cost notes — estimate_usage_cost now attaches a
'subscription-included; no provider invoice for usage' note to
CostResult for subscription-included routes (openai-codex), so
consumers can distinguish 'free because subscription' from 'free
because $0 pricing'.
Fixes#79220Fixes#77223
- Deleted the id(cfg)-keyed _OVERRIDE_CACHE layer: id() is unique only
among live objects, so a config reload could serve stale overrides
forever when CPython reuses the freed dict's address. The upstream
load_config_readonly is already (mtime,size)-cached (~1 stat/hit), so
the local layer was redundant state with a correctness risk.
- _override_to_catalog_shape returns (patch, vision) instead of
smuggling an in-band _vision_override sentinel key through the merged
dict; removed the two dead call-site pops.
- _find_model_entry gains the :cloud/-cloud suffix fallback that
lookup_models_dev_context already had, so 'catalog hit' means the
same thing to every consumer — a suffix-keyed model (kimi-k2.6:cloud)
now counts as KNOWN and keeps its catalog capabilities instead of
being displaced by a fill-gap _default (mutation-checked contract
test added).
- get_model_info's unknown-model override path seeds the same safe
defaults as get_model_capabilities (200K ctx, tools on, 8192 out),
so a partial override no longer yields ctx=0/tools-off on that path
(contract test added); the DEFAULT_CONFIG defaults claim is now true
for both paths.
- Activated the previously-dead _MODELS_DEV_TO_PROVIDER reverse map
(lazily built, many-to-one aware) and used it in
_provider_override_section instead of a per-call linear scan.
Review follow-ups on the model_overrides feature:
- ONE canonical override schema everywhere. get_model_info previously
merged the override dict raw into the models.dev catalog shape
({**raw, **override}), so the documented context_window/supports_*
keys silently did nothing on that path (cost guard, inventory) while
working in capabilities/context paths — same config key, two
incompatible schemas. Overrides are now translated into the catalog
shape at the get_model_info boundary (_override_to_catalog_shape),
and sub-dicts (limit, modalities) are MERGED, not clobbered — an
override setting only context_window no longer wipes the catalog's
limit.output.
- _default is now a FILL-GAP default, not an override: it applies only
to models the catalog does not know (the #8731/#84482 self-unblock
path) and never displaces catalog data. A
_default: {context_window: 128000} can no longer clamp every model
of a provider. Explicit per-provider+model entries keep their
win-over-catalog semantics.
- Early-chain _override_context_window (model_metadata step 0b) is
explicit-only, so a _default can never preempt custom_providers
per-model settings or live probes; fill-gap defaults apply at the
lookup_models_dev_context catalog-miss boundary (step 5f) instead.
This fixes the precedence inversion where a provider/global _default
silently overrode an explicit per-endpoint per-model context_length.
- Provider keys accept BOTH id spaces (Hermes id and models.dev id:
copilot/github-copilot both work) and model ids match
case-insensitively, mirroring catalog lookup.
- Malformed override values (context_window: '512k') log a one-shot
warning instead of being silently swallowed.
- DEFAULT_CONFIG comment: removed the false family/dated-snapshot
inheritance claim, documented the recognized field list, fill-gap
semantics, and the id-space rule.
- Tests: rewritten for the new contracts (fill-gap invariants,
dual-id-space keys, sub-dict merge preservation, one-shot warning);
added a real-config-yaml e2e plumbing test (mutation-checked: fails
when the config key wiring is broken).
Add a unified model_overrides config section that lets users manually
declare context_window, max_output_tokens, capabilities, cost, and
family for any provider+model — winning over models.dev, OpenRouter, and
hardcoded defaults.
Resolution order (first hit wins):
1. model_overrides.<provider>.<model_id> (per-provider+model)
2. model_overrides.<provider>._default (per-provider default)
3. model_overrides._default (global default)
4. Normal catalog resolution
Key subtlety: an unknown model id (not in the
catalog) derives base metadata from sensible defaults before patching,
so overriding a model the catalog doesn't know yet is the supported
self-unblock path. This is exactly the #84482 scenario (Upstage
solar-pro4/syn-pro wrong context) and the #8731 scenario (custom/local
models with manual capability declaration).
Wired into:
- get_model_capabilities() — patches capability fields; unknown models
get safe defaults (tools on, vision/reasoning off) before patching
- lookup_models_dev_context() — context_window override, checked before
catalog lookup so it works even for providers not in PROVIDER_TO_MODELS_DEV
- get_model_info() — merges override dict onto catalog entry (shallow
merge); for unknown models, the override is the sole source of metadata
- get_model_context_length() — step 0b in the resolution pipeline,
before custom_providers (0c) and before any network probe
Config example:
model_overrides:
upstage:
solar-pro4:
context_window: 524288
syn-pro:
context_window: 65536
custom:my-local-vllm:
my-llava-model:
context_window: 8192
supports_vision: true
supports_reasoning: false
supports_tools: true
_default:
context_window: 128000
Fixes#8731Fixes#84482
Refs #47247
The set is no longer in parity with agent_init's fail-fast gate (which
still skips openrouter for a different reason: default route, not
unroutable). Say so instead of claiming parity.
BARE_BILLING_PROVIDERS incorrectly included "openrouter" alongside
"auto" and "custom". OpenRouter is a fully routable provider with
its own API key and base_url — sessions that used OpenRouter store
billing_provider="openrouter", and dropping it forces resume to the
current global model (e.g. a custom endpoint), which is the wrong
provider for the stored model.
Remove "openrouter" from the bare-bucket set so OpenRouter sessions
correctly restore their provider identity on resume.
Fixes#57588
cronjob(action=create) has supported a repeat cap since the tool
existed, but the ws handler dropped the param — UIs building on
cron.manage could not create run-N-times jobs. Forward it (digit
strings accepted, None keeps schedule-kind defaults). One-shot
relative schedules (bare 30m/2h) already flow through the schedule
string untouched.
Maintainer fixup on the #79787 salvage:
- An explicit fb.api_mode of "chat_completions" was silently overridden
by the codex_responses / bedrock re-detection pass (which only skipped
re-detection when the pre-computed mode was non-default). Track
explicitness in fb_api_mode_explicit and gate the whole re-detection
block on it.
- Replace the locals().get('fb_api_mode') dead-code hack with clean code
(fb_api_mode is always bound at that point).
- Restore the post-resolve /anthropic + api.anthropic.com host check for
named custom providers whose base_url comes from config rather than
the fallback entry (#32243, #49247), which the PR's restructure dropped.
- Add regression tests: explicit api_mode honored (incl. explicit
chat_completions not overridden), /anthropic-hint fallback detected
pre-rewrite, api_mode forwarded to resolve_provider_client, plain
fallback unchanged.
Fallback activation determined api_mode from the POST-rewrite client
base_url, losing the Anthropic wire signal for /anthropic endpoints
routed through provider 'custom', and never honored an explicit
fb.api_mode config field. Pre-compute fb_api_mode from the ORIGINAL
fallback base_url hint (before _to_openai_base_url rewriting), honor
the explicit api_mode config field, check provider name before the
base_url gate, and pass api_mode into resolve_provider_client at the
fallback call site.
Salvaged from PR #79787 (chat_completion_helpers.py hunks; the
auxiliary_client.py hunk is redundant with #85466's wrap_base fix).
- /model switch now refreshes agent._custom_providers from the config
loaded during the switch before re-evaluating cache policy — a
prompt_caching flag added to config.yaml after session start was
invisible to a mid-session switch (policy read the stale init-time
snapshot while context_length resolution used the live list).
- Production-path test: real config.yaml in the modern providers: dict
shape through the real loader chain, exercising the init-order fallback
(no _custom_providers attr) for both the fable opt-in and the opus
explicit opt-out.
- Pin operator kill-switch precedence: _cache_disabled (prompt_caching.
cache_ttl falsy) beats an explicit per-model prompt_caching: true.
- Log (debug) instead of silently swallowing capability-lookup failures in
anthropic_prompt_cache_policy — a swallowed failure would otherwise
downgrade an explicit prompt_caching: true to (False, False) with zero
trace. Matches the sibling MoA branch's logger.debug style.
- Use load_config_readonly() for the None-fallback in
get_custom_provider_model_capability: the helper only reads, and the
fallback fires on the blank-stub paths (agent init before
_custom_providers is assigned, MoA/auxiliary destination planning), so
skip the ~135us defensive deepcopy per call.
- Add route-isolation regression tests at both levels (config helper +
agent policy): a prompt_caching declaration for one provider route must
never apply to another route with the same model name. Mutation-checked:
both tests fail when the URL match is disabled.
Once you have toggled a few items on and off there is no way to get
back to the shipped layout short of remembering which ids are in
STATUSBAR_HIDDEN_BY_DEFAULT. Add a row to the bar's right-click menu
that restores that set.
The row is disabled rather than hidden when nothing is customized, so
it also advertises that a shipped layout exists. Reset touches item
layout only — whole-bar visibility is a separate preference, and
resetting from the bar's own menu should not make the bar you are
right-clicking disappear.
Now that subscriptions survive `done` (completion is reversible —
on every 5s notifier tick forever. Add
kanban_db.purge_stale_done_notify_subs(): one DELETE removing subs
whose task has been done with no new events past a retention window
(age = latest task event, falling back to completed_at/created_at, so
any activity exempts the task; a reopened task is exempt by status
alone). The notifier watcher runs it per board once at startup and at
most hourly, re-reading kanban.done_sub_retention_days (config.yaml,
default 30; 0 disables) at each sweep.
cronjob(action=list) defaults to include_disabled=False, so pausing a
job makes it vanish from any ws-driven UI's next refresh — an
enable/disable toggle reads as silent deletion (reported against the
Hermes-Bot-Mode routines pane). Forward the flag from params.
Carry the worker's completion handoff into the synthetic creator wake
turn and label it as an automatic notification with inspect-the-board /
don't-recreate guidance, so a woken orchestrator doesn't re-decompose
work that already exists (#70752).
Salvaged from PR #71100 by @yinkev; ported onto the restructured wake
region (delivery_mode gating, scope_id, sub chat_id destinations). The
auto_subscribe_on_create config-default half of the original PR was
dropped as already superseded on main.
Widen #62804's class fix: _get_goal_manager_for_event and
_get_heartbeat_manager_for_event also call get_or_create_session on
behalf of the triggering event; when that event is internal the lookup
must not advance the user-activity clock either.
For a push-adapter subscription with delivery_mode='wake' the visible text
ping is intentionally skipped (the send_passive gate), so the wake injection
IS the sole delivery — yet the event cursor advanced BEFORE the wake, which
then ran best-effort with its failure swallowed. A single failed wake
permanently lost the event.
Apply the same ordering the non-push (api_server) self-post branch already
uses: attempt the wake BEFORE advancing the cursor; on failure rewind the
claim (_kanban_rewind) and bump the per-sub failure counter so the next tick
retries; on success reset the counter; drop the subscription after
MAX_SEND_FAILURES consecutive failures like text sends do. notify+wake mode
is unchanged: the text ping is the delivery and the wake stays best-effort
after the cursor advance.
Extracts the residual delivery-ordering insight from closed PR #84191.
Co-authored-by: MaximCrabbe <crabbemaxim@gmail.com>
The whole-bar visibility atom defaulted to false (opt-in). Flip it to
true so the bar shows on first launch. The context-usage meter and
other diagnostic items remain hidden via STATUSBAR_HIDDEN_BY_DEFAULT,
so only the core status items (gateway health, model pill, command
center) appear out of the box. The toggle keybind and ⌘K row still
let users hide it.
The cherry-picked tests predate #85508's honest fallback-chain phrasing
and each other: assertions pinned the old 'exhausted or unavailable'
literal and #83188's no_agent fallback-note behavior, which #77648's
mode gate supersedes (no provider classification at all for no_agent
jobs). Assert the composed contract instead.
Alex Fournier