feat: keyless web tier becomes a 5-vendor round-robin ring (adds Tavily, Firecrawl, Keenable)

Fresh installs with zero web credentials now rotate web_search/
web_extract across FIVE vendors' public free tiers — Exa, Parallel,
Tavily, Firecrawl, Keenable — instead of a 2-vendor 50/50 split, with
next-in-line ring failover on rate limits (multi-hop until a vendor
serves or the ring is exhausted; served_by marks the actual vendor).

- plugins/web/keenable/: new bundled provider (search via /v1/search,
  fetch via /v1/fetch; keyed Bearer or keyless with the mandatory
  X-Keenable-Title app header). Credit: integration proposed by
  Ilya Gusev (Keenable) in #49758; Free/Paid picker rows included.
- keyless_mcp: tavily/firecrawl/keenable keyless search+extract
  wrappers, _KEYLESS_RING + per-process round-robin cursor (seeded by
  the random session id, advances per unpinned request), pinned-vendor
  entry (pin = start there; rotation off), paid-pinned vendors excluded
  from the ring entirely.
- Tavily/Firecrawl providers route keyless traffic through the ring;
  both are now default-on ring members (no longer selection-gated).
- web_tools/registry: keenable in backend sets, auto-detect, availability
  probes; _keyless_preference() delegates to the ring cursor.
- KEENABLE_API_KEY in OPTIONAL_ENV_VARS; docs updated (ring semantics).

Live E2E: all 10 vendorXcapability paths (5 search + 5 extract) served
real results keyless; rotation cycled all five vendors over 5 dispatch
calls; double-throttle failover walked exa->parallel->tavily.
This commit is contained in:
Teknium
2026-08-20 00:17:25 -07:00
parent 02274c39dc
commit 4ea69d9d2c
15 changed files with 903 additions and 166 deletions
+5 -1
View File
@@ -169,7 +169,7 @@ def _load_web_config() -> dict:
# WebSearchProvider. Keep the two sets aligned by hand: if xai ever ships as
# a registered provider, drop it here so the registry path takes over.
_LEGACY_WEB_BACKENDS = frozenset(
{"parallel", "firecrawl", "tavily", "exa", "searxng", "brave-free", "ddgs", "xai"}
{"parallel", "firecrawl", "tavily", "exa", "searxng", "brave-free", "ddgs", "xai", "keenable"}
)
@@ -262,6 +262,7 @@ def _get_backend() -> str:
("tavily", _has_env("TAVILY_API_KEY")),
("exa", _has_env("EXA_API_KEY")),
("parallel", _has_env("PARALLEL_API_KEY")),
("keenable", _has_env("KEENABLE_API_KEY")),
("firecrawl", _has_env("FIRECRAWL_API_KEY") or _has_env("FIRECRAWL_API_URL")),
("firecrawl", _is_tool_gateway_ready()),
("searxng", _has_env("SEARXNG_URL")),
@@ -387,6 +388,8 @@ def _is_backend_available(backend: str) -> bool:
return _has_env("EXA_API_KEY")
if backend == "parallel":
return _has_env("PARALLEL_API_KEY")
if backend == "keenable":
return _has_env("KEENABLE_API_KEY")
if backend == "firecrawl":
return check_firecrawl_api_key()
if backend == "tavily":
@@ -450,6 +453,7 @@ def _web_requires_env() -> list[str]:
"EXA_API_KEY",
"PARALLEL_API_KEY",
"TAVILY_API_KEY",
"KEENABLE_API_KEY",
"FIRECRAWL_API_KEY",
"FIRECRAWL_API_URL",
"FIRECRAWL_GATEWAY_URL",