Commit Graph

8 Commits

Author SHA1 Message Date
Lakshya Agarwal 428e084dcd feat(web): add Tavily web search and extract provider
This commit re-introduces the Tavily provider, which supports both search and content extraction capabilities, which was removed in #99199.
2026-09-01 10:56:49 -07:00
Teknium d6773cf26f refactor: remove the Tavily web backend; keyless ring is exa/parallel/firecrawl/keenable
- Tavily plugin deleted (plugins/web/tavily), keyless endpoints and
  ring entry removed from keyless_mcp, legacy backend set / credential
  ladder / preference walks / rescue key map scrubbed.
- TAVILY_API_KEY deregistered across config, setup, status, dump, and
  nous_subscription surfaces. The tvly- redaction pattern stays --
  legacy keys in user envs still deserve masking.
- Sibling test pins migrated (keenable/exa stand in where tavily was
  the fixture vendor); tavily test suite deleted.
- Docs updated: web-search, configuration, integrations,
  environment-variables, tools-reference, web-dashboard, provider
  plugin dev guide.

Live-verified from an isolated HERMES_HOME with all web creds blanked:
zero-config resolution lands in the 4-vendor ring, live keyless ring
search succeeds, no tavily anywhere in resolution order.
2026-08-31 00:56:41 -07:00
Teknium e7e3885113 test: pin ring entry vendor in provider-routing tests (ring rotation made direct-callable mocks stale) 2026-08-20 00:20:17 -07:00
Teknium 4ea69d9d2c feat: keyless web tier becomes a 5-vendor round-robin ring (adds Tavily, Firecrawl, Keenable)
Fresh installs with zero web credentials now rotate web_search/
web_extract across FIVE vendors' public free tiers — Exa, Parallel,
Tavily, Firecrawl, Keenable — instead of a 2-vendor 50/50 split, with
next-in-line ring failover on rate limits (multi-hop until a vendor
serves or the ring is exhausted; served_by marks the actual vendor).

- plugins/web/keenable/: new bundled provider (search via /v1/search,
  fetch via /v1/fetch; keyed Bearer or keyless with the mandatory
  X-Keenable-Title app header). Credit: integration proposed by
  Ilya Gusev (Keenable) in #49758; Free/Paid picker rows included.
- keyless_mcp: tavily/firecrawl/keenable keyless search+extract
  wrappers, _KEYLESS_RING + per-process round-robin cursor (seeded by
  the random session id, advances per unpinned request), pinned-vendor
  entry (pin = start there; rotation off), paid-pinned vendors excluded
  from the ring entirely.
- Tavily/Firecrawl providers route keyless traffic through the ring;
  both are now default-on ring members (no longer selection-gated).
- web_tools/registry: keenable in backend sets, auto-detect, availability
  probes; _keyless_preference() delegates to the ring cursor.
- KEENABLE_API_KEY in OPTIONAL_ENV_VARS; docs updated (ring semantics).

Live E2E: all 10 vendorXcapability paths (5 search + 5 extract) served
real results keyless; rotation cycled all five vendors over 5 dispatch
calls; double-throttle failover walked exa->parallel->tavily.
2026-08-20 00:17:25 -07:00
Teknium 2eb5217af9 feat: keyless free-tier failover + Tavily/Firecrawl salvage integration
- Cross-vendor failover: when Exa's or Parallel's keyless free tier
  returns a rate-limit-shaped error, the request retries once on the
  other vendor's free endpoint (search + whole-batch extract). Result
  notes served_by; a peer pinned to its paid tier is never used;
  non-throttle errors never fail over.
- Docs: failover note + Tavily/Firecrawl keyless-when-selected rows.
- Firecrawl keyless test expectations aligned with the keyless tier.
2026-08-19 22:58:05 -07:00
Teknium 4d87290d39 feat: keyless web traffic splits 50/50 between Exa and Parallel like opencode
Unpinned zero-credential installs now pick Exa or Parallel by the
parity of the per-process random session id (stable within a process,
even split fleet-wide) instead of always favoring Parallel. An explicit
hermes tools selection (web.backend / per-capability keys) bypasses the
split entirely; the runner-up vendor stays in the walk as fallback.

Live E2E: 6 fresh processes split 3/3 between vendors, each performed
a real keyless search via its picked endpoint; explicit pin verified.
2026-08-19 16:22:30 -07:00
Teknium f08d3e400f feat: hermes tools lets Exa/Parallel users pick the free keyless or paid keyed endpoint
Exa and Parallel now each render as two picker rows in hermes tools —
'Free (keyless)' and 'Paid (API key)'. Selection persists to
web.provider_tier.<name>:
- free: always the anonymous public endpoint, even with a key set
- paid: always the keyed SDK path; missing key errors instead of
  silently downgrading to the free tier (is_keyless_available also
  returns False so the auto-fallback walk can't route there)
- unset: auto (key present -> paid, else keyless)

Mechanism: get_setup_schema() gains a 'variants' list the picker
flattens into sibling rows sharing one web_backend; selection writes
the tier via both _write_provider_config sites; active-row detection
matches the tier (auto mirrors use_keyless). Routing goes through a
single use_keyless() chokepoint shared by search+extract in both
providers.

Live E2E: tier=free with a fake key present searched keyless OK (a
keyed call would have 401'd); tier=paid without key errored naming
PARALLEL_API_KEY; picker rows verified for both vendors x both tiers.
2026-08-19 15:36:20 -07:00
Teknium 96c2fd3c04 feat: web search/extract now work keyless on fresh installs via Parallel + Exa free tiers
With zero web credentials configured, web_search/web_extract previously
resolved to the nonfunctional firecrawl sentinel and errored. Now the
backend resolution walks a strictly-last keyless tier: Parallel's and
Exa's public anonymous MCP endpoints (the same free tiers opencode ships
as its default search path).

- plugins/web/keyless_mcp.py: minimal JSON-RPC tools/call client for
  mcp.exa.ai + search.parallel.ai (SSE + plain JSON parsing, typed
  errors, per-process random session id, no user identifiers)
- WebSearchProvider.is_keyless_available(): separate weaker tier that
  never leaks into is_available(), so keyed setups are never pre-empted
- Exa/Parallel providers: route to keyless endpoints when their key is
  absent; keyed SDK path unchanged
- registry + _get_backend(): keyless walk (parallel -> exa) strictly
  after every keyed/importable candidate; check_web_api_key() lights
  the tools up on zero-credential installs
- web.keyless_fallback config key (default true) to disable the tier
- docs: web-search.md + configuration.md

E2E-verified against both live endpoints from an isolated HERMES_HOME
(search + extract via the real dispatchers, disable-flag negative path).
2026-08-19 15:15:27 -07:00