- Tavily plugin deleted (plugins/web/tavily), keyless endpoints and
ring entry removed from keyless_mcp, legacy backend set / credential
ladder / preference walks / rescue key map scrubbed.
- TAVILY_API_KEY deregistered across config, setup, status, dump, and
nous_subscription surfaces. The tvly- redaction pattern stays --
legacy keys in user envs still deserve masking.
- Sibling test pins migrated (keenable/exa stand in where tavily was
the fixture vendor); tavily test suite deleted.
- Docs updated: web-search, configuration, integrations,
environment-variables, tools-reference, web-dashboard, provider
plugin dev guide.
Live-verified from an isolated HERMES_HOME with all web creds blanked:
zero-config resolution lands in the 4-vendor ring, live keyless ring
search succeeds, no tavily anywhere in resolution order.
Sites under active development but tested over the public internet
(Vercel previews, ngrok tunnels, staging domains) are public DNS, so
the local-dev never-cache rule can't catch them. web.cache_exempt_hosts
lists hosts whose pages are always fetched live: exact, "*.wildcard",
or domain-suffix matching (label-boundary aware — mysite.dev covers
preview.mysite.dev but never evilmysite.dev). Checked on both store
and lookup, so adding an exemption takes effect immediately even for
entries cached before the config change.
Dev servers, hot-reload builds, and chat-GUI artifact previews live on
localhost/private addresses and change on every save — a 20-minute
cached copy would show a stale build exactly when freshness is the
point of fetching. The extract cache now declines loopback, private,
link-local, *.local, *.localhost, and single-label LAN hostnames on
both put and get. Hostname heuristics only (no DNS) — this is a
freshness carveout, not a security boundary; SSRF enforcement is
unchanged in tools/url_safety.py.
Public URLs keep the full TTL.
Review fixes for #94618 (all three blockers reproduced by the reviewer
through the real web_extract_tool):
1. Cache lookup moved AFTER provider resolution and strict-selection
validation, and gated per-URL on the website blocklist policy — a
blocklist-blocked or misconfigured-backend call now behaves exactly
as it would without a cache instead of serving cached content.
2. Rescue-served extract batches are never cached (mirrors the search
memo's exclusion), keeping one-shot rescue one-shot.
3. Cache entries now get dedicated per-(url, format, provider) files
instead of sharing the URL-keyed truncate-store file — html and
markdown (or two backends') copies of one URL no longer overwrite
each other, and switching extract backends within the TTL never
serves the old backend's rendering.
Also from review: per-process index tmp filename (cross-process writers
can no longer truncate each other mid-write) and held flight locks are
never evicted from the bounded lock table (eviction could have allowed
a duplicate paid request).
New regression tests for formats/provider keying; E2E harness extended
with policy-block, strict-selection, rescue-two-call, and dual-format
scenarios — 6/6 pass; original 13/13 still pass.
Repeat searches (same normalized query + provider) within a 20-minute
TTL are served from an in-process memo, and concurrent identical
queries are single-flighted so a parallel subagent fan-out pays for
one vendor request instead of N. Requested limits bucket up to
10/20/50/100 so near-identical requests share an entry; callers get
their requested count sliced from the bucket.
Repeat extracts of the same URL are served from the existing
cache/web full-text store (previously written for read_file paging
but never read back), via a small JSON sidecar index. Disk-backed, so
CLI, gateway, cron, and subagents share it. Cached extracts re-run
the normal truncate pipeline, so per-call char_limit still works.
Both caches sit after every safety gate (secret-URL, SSRF, policy,
provider resolution) and directly around the paid vendor call — hits
skip only the network request. Only successful responses cache;
rescue-served responses are never cached (one-shot rescue must stay
one-shot). Config: web.cache_enabled (default on),
web.cache_ttl_minutes (default 20, clamped 1-1440).
Idea credit: query coalescing + num-bucketing pattern observed in
Apodex FrontierAgent (Apache-2.0).
When the chosen/keyed backend fails a web_search or web_extract call
(bad key, upstream outage, 5xx, raised exception), that single call
retries on the keyless free-tier ring instead of erroring. The next
call attempts the chosen backend again — no sticky failover, no state.
Resolves the keyed half of #78984/#32159 (keyless half landed in the
ring PR).
- tools/web_tools.py: _rescue_eligible (keyed ring vendors + non-ring
backends eligible; keyless-mode calls excluded — they already walked
the ring), _rescue_search/_rescue_extract (search annotates
rescued_from + backend_error naming the original failure and the
retry-next-call semantics; extract rescues only whole-batch failures,
partial failures pass through untouched; rescue failure preserves the
ORIGINAL backend error with the rescue note appended)
- both dispatchers wrap the provider call: failure-results AND raised
exceptions rescue; ineligible paths re-raise unchanged
- web.keyless_rescue config key (default true; implicitly off when
keyless_fallback is off); docs updated
Live E2E: keyed Tavily with an invalid key 401'd and the call was
served by the real ring with the rescue annotation; a second call
re-attempted Tavily first (statelessness proven); whole-batch extract
rescue returned real page content. 13 new tests; 67 green across the
keyless suites.
Fresh installs with zero web credentials now rotate web_search/
web_extract across FIVE vendors' public free tiers — Exa, Parallel,
Tavily, Firecrawl, Keenable — instead of a 2-vendor 50/50 split, with
next-in-line ring failover on rate limits (multi-hop until a vendor
serves or the ring is exhausted; served_by marks the actual vendor).
- plugins/web/keenable/: new bundled provider (search via /v1/search,
fetch via /v1/fetch; keyed Bearer or keyless with the mandatory
X-Keenable-Title app header). Credit: integration proposed by
Ilya Gusev (Keenable) in #49758; Free/Paid picker rows included.
- keyless_mcp: tavily/firecrawl/keenable keyless search+extract
wrappers, _KEYLESS_RING + per-process round-robin cursor (seeded by
the random session id, advances per unpinned request), pinned-vendor
entry (pin = start there; rotation off), paid-pinned vendors excluded
from the ring entirely.
- Tavily/Firecrawl providers route keyless traffic through the ring;
both are now default-on ring members (no longer selection-gated).
- web_tools/registry: keenable in backend sets, auto-detect, availability
probes; _keyless_preference() delegates to the ring cursor.
- KEENABLE_API_KEY in OPTIONAL_ENV_VARS; docs updated (ring semantics).
Live E2E: all 10 vendorXcapability paths (5 search + 5 extract) served
real results keyless; rotation cycled all five vendors over 5 dispatch
calls; double-throttle failover walked exa->parallel->tavily.
- Cross-vendor failover: when Exa's or Parallel's keyless free tier
returns a rate-limit-shaped error, the request retries once on the
other vendor's free endpoint (search + whole-batch extract). Result
notes served_by; a peer pinned to its paid tier is never used;
non-throttle errors never fail over.
- Docs: failover note + Tavily/Firecrawl keyless-when-selected rows.
- Firecrawl keyless test expectations aligned with the keyless tier.
- Updated the Tavily API key description to clarify that it is optional and keyless access is supported.
- Modified the Tavily plugin and provider to handle requests with or without an API key, using Bearer authentication when the key is provided.
- Enhanced documentation to reflect the new keyless functionality and updated environment variable descriptions.
- Added tests to ensure correct behavior for both keyed and keyless requests.
Unpinned zero-credential installs now pick Exa or Parallel by the
parity of the per-process random session id (stable within a process,
even split fleet-wide) instead of always favoring Parallel. An explicit
hermes tools selection (web.backend / per-capability keys) bypasses the
split entirely; the runner-up vendor stays in the walk as fallback.
Live E2E: 6 fresh processes split 3/3 between vendors, each performed
a real keyless search via its picked endpoint; explicit pin verified.
Exa and Parallel now each render as two picker rows in hermes tools —
'Free (keyless)' and 'Paid (API key)'. Selection persists to
web.provider_tier.<name>:
- free: always the anonymous public endpoint, even with a key set
- paid: always the keyed SDK path; missing key errors instead of
silently downgrading to the free tier (is_keyless_available also
returns False so the auto-fallback walk can't route there)
- unset: auto (key present -> paid, else keyless)
Mechanism: get_setup_schema() gains a 'variants' list the picker
flattens into sibling rows sharing one web_backend; selection writes
the tier via both _write_provider_config sites; active-row detection
matches the tier (auto mirrors use_keyless). Routing goes through a
single use_keyless() chokepoint shared by search+extract in both
providers.
Live E2E: tier=free with a fake key present searched keyless OK (a
keyed call would have 401'd); tier=paid without key errored naming
PARALLEL_API_KEY; picker rows verified for both vendors x both tiers.
A 12-request sequential burst from the same IP that earlier saw the
free-tier rate-limit error went 12/12 OK — the limit is a transient
burst/load control, not a tight standing per-IP quota. Soften the docs
and setup-schema wording accordingly (opencode users hit Exa keyless
as their default path in practice without throttling).
With zero web credentials configured, web_search/web_extract previously
resolved to the nonfunctional firecrawl sentinel and errored. Now the
backend resolution walks a strictly-last keyless tier: Parallel's and
Exa's public anonymous MCP endpoints (the same free tiers opencode ships
as its default search path).
- plugins/web/keyless_mcp.py: minimal JSON-RPC tools/call client for
mcp.exa.ai + search.parallel.ai (SSE + plain JSON parsing, typed
errors, per-process random session id, no user identifiers)
- WebSearchProvider.is_keyless_available(): separate weaker tier that
never leaks into is_available(), so keyed setups are never pre-empted
- Exa/Parallel providers: route to keyless endpoints when their key is
absent; keyed SDK path unchanged
- registry + _get_backend(): keyless walk (parallel -> exa) strictly
after every keyed/importable candidate; check_web_api_key() lights
the tools up on zero-credential installs
- web.keyless_fallback config key (default true) to disable the tier
- docs: web-search.md + configuration.md
E2E-verified against both live endpoints from an isolated HERMES_HOME
(search + extract via the real dispatchers, disable-flag negative path).
Cross-checked website/docs against the source at main HEAD and corrected
documented commands, env vars, config keys, headers, and default values
that don't match the code. Docs-only; no behavioral changes.
Refs #36048
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflect the default-model change in the xAI Grok OAuth guide, the web
search docs (EN + zh-Hans), and the web provider docstring. grok-4.3 is
kept in the model tables as the previous default; the Nous/OpenRouter
aggregator catalog still lists grok-4.3 and is left unchanged.
The web_crawl_tool() function was an orphan — no model schema registered
it, no skill or CLI command called it, and the agent had no way to invoke
it. PR #32608 proposed wiring it up as a model-callable tool; we've
decided not to expose crawl as a separate capability since web_search +
web_extract cover the use cases we want models to have.
Removed:
- tools/web_tools.py: web_crawl_tool() (~230 LOC)
- plugins/web/firecrawl/provider.py: supports_crawl() + crawl()
- plugins/web/tavily/provider.py: supports_crawl() + crawl()
- plugins/web/xai/provider.py: supports_crawl() override
- agent/web_search_provider.py: supports_crawl() + crawl() ABC methods
- agent/web_search_registry.py: get_active_crawl_provider() +
the 'crawl' branch in _resolve()
- agent/display.py: web_crawl tool-progress rendering
- hermes_cli/config.py: 'web_crawl' from TAVILY_API_KEY.tools
- tools/website_policy.py: stale comment reference
- Tests: removed TestWebCrawlTavily class, the two website-policy
web_crawl tests, the searxng/ddgs/brave-free crawl-error tests,
the integration test_web_crawl method, and the
test_unconfigured_crawl_emits_top_level_error test. Trimmed the
capability-flag parametrize list and the WebSearchProvider ABC
conformance tests.
- Docs: trimmed the Crawl column from capability tables in both EN
and zh-Hans, updated the developer-guide ABC table.
Net: 25 files, +115/-1067.
Closes#33762 (the schema-text bug only existed if #32608 landed).
Supersedes #32608.
The locale switcher appeared broken because hardcoded markdown links
(`](/docs/X)`) got double-prefixed by Docusaurus to `/docs/<locale>/docs/X`
(404) in non-English locales, and the MDX hero `<a href>` on the index
page escaped locale routing entirely.
Changes:
- Rewrite 922 `](/docs/X)` -> `](/X)` across 166 docs files (strip trailing
.md too). Docusaurus prepends locale + baseUrl itself.
- docs/index.md -> index.mdx; hero "Get Started" anchor -> Docusaurus
<Link> so it stays inside the active locale.
- Drop `ko` locale entirely from docusaurus.config.ts + delete i18n/ko/
(4 stale auto-translated kanban pages, <2% coverage, misleading).
Verified `npm run build` succeeds for both en and zh-Hans; `build/zh-Hans/
index.html` has no /docs/zh-Hans/docs/... double-prefixed paths.
PR2 will translate the 335 English docs into i18n/zh-Hans/.
PR #30860 added a one-shot Portal setup command and a small portal CLI
surface. Update the docs so the new commands are discoverable without
upgrading the tone of existing Portal mentions.
- getting-started/quickstart.md: small tip near Choose a Provider
pointing at 'hermes setup --portal' as the easiest fresh-install path.
- user-guide/features/tool-gateway.md: lead the Get-Started section
with 'hermes setup --portal' for fresh installs, keep 'hermes model'
for already-configured users, and add 'hermes portal status / tools'
to the activity-check commands.
- user-guide/features/{web-search,image-generation,tts,browser}.md: the
existing 'Nous Subscribers' tip blocks now name the one-shot command
for new installs, keeping the existing 'hermes tools' path for users
who only want to swap a single backend.
- reference/cli-commands.md: register 'hermes portal' in the top-level
command table, add a 'hermes portal' section with subcommands, and
add '--portal' to the 'hermes setup' options table.
Tone: each page already had a Portal mention. This PR keeps the per-page
count to one and uses concrete CLI commands rather than promotional copy.
Tool Gateway page is the one exception (the whole doc is about Portal).
Follow-up to #29042 (xAI Web Search provider plugin). Adds xAI to the
canonical user-facing and developer-facing docs, with the search-only
caveat and the LLM-in-a-trench-coat trust model carried over from the
class docstring.
- user-guide/features/web-search.md
- Backends table: new xAI row + extended search-only note
- New 'xAI (Grok)' setup section with config knobs and trust-model
caution admonition
- Single-backend yaml comment now lists 'xai'
- Auto-detection table: explicitly note that xAI is NOT auto-detected
(XAI_API_KEY is shared with inference/TTS/image-gen so we don't
silently take over web for users who only set it for chat)
- developer-guide/web-search-provider-plugin.md
- Added plugins/web/xai/ to the 'study these next' reference list
- reference/environment-variables.md
- XAI_API_KEY description now also mentions web search
web_extract runs returned page content through the web_extract auxiliary
model when pages exceed 5 000 chars (single-pass up to 500k, chunked up
to 2M, refused above that). The user-guide page didn't mention this —
users were surprised that long-page extracts produced summaries instead
of raw markdown, and that those summaries cost main-model tokens by
default.
Adds:
- size-driven behavior table (under 5k / 5k–500k / 500k–2M / over 2M)
- which auxiliary task does the work (auxiliary.web_extract)
- how to route summaries to a cheap model regardless of main
- escape hatch: browser_navigate when you need raw content
- troubleshooting entry for summarization timeouts
* docs: deep audit — fix stale config keys, missing commands, and registry drift
Cross-checked ~80 high-impact docs pages (getting-started, reference, top-level
user-guide, user-guide/features) against the live registries:
hermes_cli/commands.py COMMAND_REGISTRY (slash commands)
hermes_cli/auth.py PROVIDER_REGISTRY (providers)
hermes_cli/config.py DEFAULT_CONFIG (config keys)
toolsets.py TOOLSETS (toolsets)
tools/registry.py get_all_tool_names() (tools)
python -m hermes_cli.main <subcmd> --help (CLI args)
reference/
- cli-commands.md: drop duplicate hermes fallback row + duplicate section,
add stepfun/lmstudio to --provider enum, expand auth/mcp/curator subcommand
lists to match --help output (status/logout/spotify, login, archive/prune/
list-archived).
- slash-commands.md: add missing /sessions and /reload-skills entries +
correct the cross-platform Notes line.
- tools-reference.md: drop bogus '68 tools' headline, drop fictional
'browser-cdp toolset' (these tools live in 'browser' and are runtime-gated),
add missing 'kanban' and 'video' toolset sections, fix MCP example to use
the real mcp_<server>_<tool> prefix.
- toolsets-reference.md: list browser_cdp/browser_dialog inside the 'browser'
row, add missing 'kanban' and 'video' toolset rows, drop the stale
'38 tools' count for hermes-cli.
- profile-commands.md: add missing install/update/info subcommands, document
fish completion.
- environment-variables.md: dedupe GMI_API_KEY/GMI_BASE_URL rows (kept the
one with the correct gmi-serving.com default).
- faq.md: Anthropic/Google/OpenAI examples — direct providers exist (not just
via OpenRouter), refresh the OpenAI model list.
getting-started/
- installation.md: PortableGit (not MinGit) is what the Windows installer
fetches; document the 32-bit MinGit fallback.
- installation.md / termux.md: installer prefers .[termux-all] then falls
back to .[termux].
- nix-setup.md: Python 3.12 (not 3.11), Node.js 22 (not 20); fix invalid
'nix flake update --flake' invocation.
- updating.md: 'hermes backup restore --state pre-update' doesn't exist —
point at the snapshot/quick-snapshot flow; correct config key
'updates.pre_update_backup' (was 'update.backup').
user-guide/
- configuration.md: api_max_retries default 3 (not 2); display.runtime_footer
is the real key (not display.runtime_metadata_footer); checkpoints defaults
enabled=false / max_snapshots=20 (not true / 50).
- configuring-models.md: 'hermes model list' / 'hermes model set ...' don't
exist — hermes model is interactive only.
- tui.md: busy_indicator -> tui_status_indicator with values
kaomoji|emoji|unicode|ascii (not kawaii|minimal|dots|wings|none).
- security.md: SSH backend keys (TERMINAL_SSH_HOST/USER/KEY) live in .env,
not config.yaml.
- windows-wsl-quickstart.md: there is no 'hermes api' subcommand — the
OpenAI-compatible API server runs inside hermes gateway.
user-guide/features/
- computer-use.md: approvals.mode (not security.approval_level); fix broken
./browser-use.md link to ./browser.md.
- fallback-providers.md: top-level fallback_providers (not
model.fallback_providers); the picker is subcommand-based, not modal.
- api-server.md: API_SERVER_* are env vars — write to per-profile .env,
not 'hermes config set' which targets YAML.
- web-search.md: drop web_crawl as a registered tool (it isn't); deep-crawl
modes are exposed through web_extract.
- kanban.md: failure_limit default is 2, not '~5'.
- plugins.md: drop hard-coded '33 providers' count.
- honcho.md: fix unclosed quote in echo HONCHO_API_KEY snippet; document
that 'hermes honcho' subcommand is gated on memory.provider=honcho;
reconcile subcommand list with actual --help output.
- memory-providers.md: legacy 'hermes honcho setup' redirect documented.
Verified via 'npm run build' — site builds cleanly; broken-link count went
from 149 to 146 (no regressions, fixed a few in passing).
* docs: round 2 audit fixes + regenerate skill catalogs
Follow-up to the previous commit on this branch:
Round 2 manual fixes:
- quickstart.md: KIMI_CODING_API_KEY mentioned alongside KIMI_API_KEY;
voice-mode and ACP install commands rewritten — bare 'pip install ...'
doesn't work for curl-installed setups (no pip on PATH, not in repo
dir); replaced with 'cd ~/.hermes/hermes-agent && uv pip install -e
".[voice]"'. ACP already ships in [all] so the curl install includes it.
- cli.md / configuration.md: 'auxiliary.compression.model' shown as
'google/gemini-3-flash-preview' (the doc's own claimed default);
actual default is empty (= use main model). Reworded as 'leave empty
(default) or pin a cheap model'.
- built-in-plugins.md: added the bundled 'kanban/dashboard' plugin row
that was missing from the table.
Regenerated skill catalogs:
- ran website/scripts/generate-skill-docs.py to refresh all 163 per-skill
pages and both reference catalogs (skills-catalog.md,
optional-skills-catalog.md). This adds the entries that were genuinely
missing — productivity/teams-meeting-pipeline (bundled),
optional/finance/* (entire category — 7 skills:
3-statement-model, comps-analysis, dcf-model, excel-author, lbo-model,
merger-model, pptx-author), creative/hyperframes,
creative/kanban-video-orchestrator, devops/watchers,
productivity/shop-app, research/searxng-search,
apple/macos-computer-use — and rewrites every other per-skill page from
the current SKILL.md. Most diffs are tiny (one line of refreshed
metadata).
Validation:
- 'npm run build' succeeded.
- Broken-link count moved 146 -> 155 — the +9 are zh-Hans translation
shells that lag every newly-added skill page (pre-existing pattern).
No regressions on any en/ page.
Lakshya Agarwal