Adds two bounded fast modes on top of the static /fast toggle, default OFF:
- `auto`: every user turn opens a `agent.fast_auto_seconds` (default 60s)
window; requests inside it carry the provider fast param, later tool-loop
requests fall back to standard pricing.
- `cold`: the same window, but only on the first turn of a session (no prior
user/assistant/tool history).
agent/fast_mode.py holds the whole policy: `begin_turn()` at the
run_conversation ingress arms `agent._fast_until`; `effective_request_overrides()`
is consumed in the ONE place request_overrides feed the transports
(build_api_kwargs), so the fast param is a per-request kwarg only. System
prompt, tools and messages are untouched — the prompt cache is preserved.
resolve_fast_mode_overrides() is now the single gate for static and bounded
modes and accepts provider/base_url: OpenRouter, Nous, Copilot, Azure,
Bedrock and custom base_urls never receive service_tier/speed (#34308's
route gating). Both existing callers (CLI turn route, gateway turn route)
and the TUI config.set path pass the route.
Surfaces: config `agent.service_tier: auto|cold` + `agent.fast_auto_seconds`,
`/fast auto|cold` in CLI, gateway (picker gains both entries), TUI/desktop
config.set; status shows the mode; web dashboard select lists the real
values. Docs: configuration.md Fast Mode section with mode table + cost note,
slash-commands, cli-config.yaml.example, locale strings for the two picker
entries.
Salvages #89991 (bounded fast modes) and #34308 (route gating).
Fixes#64785, #74730.
Co-authored-by: Eva <239388517+100yenadmin@users.noreply.github.com>
Co-authored-by: kbaicai <kbaicai@qq.com>
Follow-up to the previous commit (which fixed the default/fallback
provider path). A mid-session `/model` switch stores a per-session
override bundle in `_session_model_overrides` that omitted
`request_overrides`, and the two consumers
(`_resolve_session_agent_runtime` fast path and
`_apply_session_model_override`) only copied
provider/api_key/base_url/api_mode. So switching *to* a custom provider
via `/model` did not apply its `extra_body`.
- `ModelSwitchResult` gains a `request_overrides` field, derived for the
switched provider via `_get_named_custom_provider` /
`_custom_provider_request_overrides` (the same overrides
`resolve_runtime_provider` surfaces for the default path).
- Both `/model` override-storage sites in slash_commands.py persist it.
- Both consumers apply it; `_apply_session_model_override` also clears a
stale value when switching to a provider that has none.
Extends tests/gateway/test_turn_request_overrides.py (3 new cases).
A `custom_providers` entry can carry an `extra_body` (e.g.
`chat_template_kwargs` to toggle a local vLLM model's thinking).
`resolve_runtime_provider()` correctly surfaces it as `request_overrides`
on the resolved runtime dict, but the gateway never plumbed it through to
the per-turn agent:
- `_resolve_runtime_agent_kwargs()` rebuilt the runtime dict from a fixed
key whitelist that omitted `request_overrides`.
- `_resolve_turn_agent_config()` rebuilt `runtime` from the same whitelist
and set `route["request_overrides"]` solely from `/fast` service-tier
overrides (`{}` otherwise).
- The per-turn `agent.request_overrides = turn_route.get(...)` assignment
then clobbered the value `_merge_custom_provider_extra_body()` applied at
agent construction.
Net: on the gateway, a custom provider's configured `extra_body` never
reached the model -- only `/fast` overrides survived. The CLI/TUI path
(which does not go through `_resolve_turn_agent_config`) and the auxiliary
client (which sends `extra_body` directly) were unaffected.
Fix: carry `request_overrides` through the runtime resolvers
(`_resolve_runtime_agent_kwargs`, `_try_resolve_fallback_provider`) and
merge the provider overrides into the per-turn route, layering any `/fast`
service-tier overrides on top (top-level keys, no collision with
`extra_body`).
Adds tests/gateway/test_turn_request_overrides.py.
Known follow-up: the mid-session `/model`-switch override path
(`_session_model_overrides` / `ModelSwitchResult`) does not yet carry
`request_overrides`.
Teknium