3 Commits

Author	SHA1	Message	Date
Austin Pickett	e0c3caf3b8	fix(model-picker): serve cached custom-provider catalog on no-probe opens (supersedes #81665, #81556) (#81973) ... * fix(model-picker): serve cached custom-provider catalog on no-probe opens #58183 stopped GUI picker opens from live-probing saved custom OpenAI-compatible endpoints so a stopped local server could not stall the picker. It gated the whole discovery block, not just the network call, so `cached_fetch_api_models()` was skipped too — and with it the catalog an earlier probe had already written to `provider_models_cache.json`. A custom endpoint that is not the current provider therefore renders only the models named in its config entry. A local server with 8 models loaded shows the 1 model that was saved when the provider was first added, on every picker open, while an explicit Refresh shows all 8. Add `cache_only` to `cached_fetch_api_models()`: answer from disk within the existing stale-serve window, never fetch, never revalidate off-thread, return None on a miss. Split the three call sites in `list_authenticated_providers()` into what the user's config permits (`discover_models`, an explicit `models:` allowlist) and how we may obtain it, so suppressing the probe now downgrades to a cached read instead of skipping discovery outright. `discover_models: false` still pins, and a cache hit no longer writes back to config since the probe that populated it already did. The latency win stands: a cold cache is a miss, so picker opens against offline endpoints still make zero network calls. * test(model-picker): pin the cached-catalog contract for no-probe opens Cover both halves of the invariant, since fixing either one alone reintroduces a bug the other guards against. `cache_only` on `cached_fetch_api_models()`: a fresh entry and an entry past its TTL but inside the stale-serve window both serve; an entry beyond that window, an empty cache, rotated credentials, `force_refresh`, and a missing base_url are all misses — and none of them fetch or spawn a background revalidation. `list_authenticated_providers()` on the GUI path: a non-current endpoint with a warm cache reports its full catalog across all three provider shapes (`custom_providers`, `providers:`, bare `provider: custom`) with no live fetch attempted. A cold cache keeps the configured list and still makes no network call, which is the #58183 guarantee. `discover_models: false` keeps pinning, and a cache hit does not write back to config. * fix: persist discovered custom-provider models in the hermes model flow The `hermes model` named-custom-provider flow (_model_flow_named_custom) probes the endpoint and shows the full catalog, but never persists it to the entry's `models:` list. No-probe surfaces (dashboard, desktop, ACP) call build_models_payload(..., probe_custom_providers=False) and only render the configured `models:` list, so a provider added via `hermes model` collapses to the single `model:` default everywhere except the CLI. OpenAI-compatible providers added via a probing picker already benefit from _save_discovered_models_to_config; the CLI flow did not. Persist the live catalog after a successful probe, mirroring the picker path in model_switch.py. A failed save is non-fatal. * fix(model-picker): stop an auto-saved catalog pinning a keyless endpoint The cached-catalog read added for no-probe picker opens still sat behind the no-key discovery gate, so it never reached the shape that motivated it: a keyless local model server. `bool(api_key) or not has_explicit_models` is a network-cost gate. It exists so Hermes does not probe an endpoint it cannot authenticate to when that endpoint already declares its catalog (5f00f36ba, 1039e90b5). Reading a catalog an earlier probe already paid for costs nothing, so the gate belongs on the probe, not on discovery as a whole. Left on the discovery side it re-pins the endpoint it was meant to spare. A successful probe calls `_save_discovered_models_to_config()`, which writes a plain list into `models:` — exactly the shape `_models_config_is_allowlist()` reads back as an explicit user allowlist. A keyless server therefore froze on the catalog of its first probe and could never widen again, which is the "lineup changes after config was written" case. f66319097 already carved the dict shape out of this trap for the same reason; the list shape is the other door into it. Move the clause to `_probe_live` at both custom-endpoint sites. Probe suppression is unchanged — verified byte-identical to main across the keyed/keyless x declared/undeclared matrix — and `discover_models: false` remains the documented way to pin a catalog. * test(model-picker): cover the keyless auto-save pinning trap Three tests around the gate move, each failing on the code before it: - a keyless endpoint carrying an auto-saved `models:` list still reads its full cached catalog - the same row, cold cache and probing enabled, still makes zero live fetches — the network-cost gate the clause exists for - an end-to-end round trip: persist a probe result via `_save_discovered_models_to_config()`, reload it, and assert the shape we wrote does not read back as a user pin The round-trip test guards the whole chain rather than one branch, so a future change that makes the saved shape look like an intentional allowlist fails here even if the gate logic is refactored. * fix(model-picker): key the custom-endpoint model cache by api_mode `cached_fetch_api_models()` fingerprints entries with `api_mode`, but no call site in `list_authenticated_providers()` passed it, so every custom row resolved to the `api_mode=None` fingerprint. Two rows sharing a base_url and credential but differing by `api_mode` are deliberately distinct picker rows — it is part of `group_key` at both sites — yet they collapsed onto one cache entry. That was latent while probing was the only way to fill a row: a mismatched entry was overwritten by the row's own live fetch. Serving that entry without a probe makes it visible, so an `anthropic_messages` row could render the catalog an OpenAI-mode row cached against the same URL. The wire protocols differ (`x-api-key` + `anthropic-version` vs `Authorization: Bearer`), so those catalogs are not interchangeable. Persist `api_mode` on the group at both grouping sites — it is already part of `group_key`, so it is constant across the group — and pass it into the cache read. Section 3b (bare `provider: custom`) has no `api_mode` in scope and already reads with the empty-credential fingerprint, so it is unchanged. Reported by Copilot review on #81973. --------- Co-authored-by: xxxigm <tuancanhnguyen706@gmail.com> Co-authored-by: Navlem <114683850+Navlem@users.noreply.github.com>	2026-08-08 16:07:03 -04:00
kshitij	7cf71c32bb	fix: follow-ups for salvaged PR #80740 ... - Give cached_fetch_api_models the same stale-while-revalidate tier as cached_provider_model_ids: TTL-expired entries within the 7d window are served instantly while a background refresh rewrites the cache — without this, every /model open an hour into the session re-blocked on the live probe (#72762's stall class, deferred). - Generalize _spawn_swr_refresh(cache_key, refresh_fn) so non-slug custom:<base_url> keys reuse the same inflight-dedupe scaffolding; slug behavior unchanged (default refresh_fn preserved). - Convert the missed sibling site: acp_adapter/server.py _named_custom_provider_catalogs() live-probed every custom_providers row's /v1/models per ACP catalog build. - Extract _cache_entry_valid() (the fp/models predicate existed 4x) and validate 'at' is numeric so hand-edited/corrupt cache JSON degrades to a live fetch instead of raising through the picker's blanket except. - Flatten the dead api_mode conditional (fetch_api_models declares api_mode=None; branch was behaviorally inert). - Tests: 4 new guards (stale-serve, stale-window cutoff, generalized SWR write-through, corrupt-at degradation) — stale-serve and corrupt-at mutation-checked; 2 existing tests updated for the new behavior.	2026-08-07 21:02:40 +05:30
Prashant Jain	fb435aae97	perf(model): disk-cache custom-provider /v1/models probes ... Custom OpenAI-compatible endpoints (named custom_providers rows, bare provider: custom, and per-endpoint-map entries) called fetch_api_models() directly at three call sites in model_switch.py, with no disk cache — unlike first-class providers, which go through cached_provider_model_ids(). Every plain /model open live-probed the active custom endpoint's /v1/models, regardless of how recently it had already been probed. Adds cached_fetch_api_models() in hermes_cli/models.py: a TTL disk-cache wrapper keyed on custom:<base_url> (custom endpoints have no PROVIDER_REGISTRY slug to key on) and fingerprinted on api_key/api_mode/ headers, with the same stale-beats-nothing fallback policy as cached_provider_model_ids(). Routes all three probe call sites through it. Since prewarm_picker_cache_async() already calls list_authenticated_providers() with probe_custom_providers defaulting True, this also fixes the endpoint being warmed on boot (populating the disk cache) instead of that work being discarded on every open — any custom endpoint (an LLM gateway, a self-hosted vLLM/SGLang server, etc.), not just one specific provider. Fixes #72762. Salvaged from #72810 per review feedback: extracts just the verified custom-endpoint cache fix with real cache-contract test coverage (hit/stale/rotation/refresh/fallback), leaving the credential-pool and Copilot-token-exchange costs described in the issue for separate follow-up.	2026-08-07 21:02:40 +05:30