Siddharth Balyan
ee2f5629b8
* refactor(connectors): cut comments that restate the code
Connector modules (tools/connectors, tui_gateway connector RPCs, desktop
connector card/store) keep only comments that carry a non-derivable why or
a cross-module contract. No behaviour change.
* feat(connectors): managed connect runs on the connection operation
Managed `connect` / `reconnect` mint one ConnectionOperation for every target and, on a
desktop session, block the tool turn until the operation settles; the result is per-target
outcomes and never carries a connect link. Off the desktop the result carries the links and
returns at once (PR3 delivers them as their own message).
Why: the previous leg handed the model a URL and a `wait` verb, and the renderer ran its own
2s poller on top of the backend's 5s one; both walked the whole gateway catalog at two vendor
calls per page to read one row (~3 Composio calls/s per pending target). A hidden composer
message started the model's `wait` on the user's behalf. None of it was observable from the
operation the MCP leg already used.
What the operation looks like now:
- `contract.py`: TargetState / Actor / SettleReason enums and the `(kind, from) -> {to: actor}`
transition table. `operation.transition()` enforces it; a card cannot claim a managed
target `connected`, only the backend watcher can.
- `live.py`: one open operation per session, found by `op_id`. `connectors.operation.status`
reads it, `connection.respond` drives it, `pending_connection` on resume replays it.
- `run.py`: the one lifecycle for both target kinds (prepare -> card -> wake/observe loop ->
settle -> result). The managed `observe` hook polls the gateway list once per tick for the
whole operation; the exact-status route replaces that call when the gateway ships it.
- `connection.update` is emitted on every transition and on settlement; registered in the
shared event contract with the operation vocabulary typed on the TS side.
- `wait`, `_rendered_links`, `_seen_instructions`, the just-minted bounce and `_clamp_timeout`
are deleted. `force` on `reconnect` always reinitiates; plain `reconnect` repairs only what
the gateway reports disconnected.
- `connections.wait_timeout_seconds` is removed from config defaults, the example and the
docs. The deadline is `OPERATION_DEADLINE_SECONDS = 300` in `operation.py`; the key was
added on this unmerged train so no migration is needed.
- Wire model: `statusReason` parsed on connection results; the seven-state `connectionStatus`
is typed on list items and an unknown value fails validation; `CONNECTION_REQUIRED` carries
`connect_card_available` instead of the link when the session platform is `desktop`.
Session platform, not callback presence, decides whether a card exists: the GUI bridge
attaches callbacks to every backend session, terminal TUI included.
* feat(desktop): connector card subscribes to the connection operation
The card renders from the backend's operation instead of driving its own: `connector-flow.ts`
(the renderer's 2s `connectors.list` poller, its 120s client deadline and `keepWaiting`) is
deleted, and both hidden composer submits in `connector-tool.tsx` go with it. The model is
never nudged into a `wait`; the tool call is blocked on the backend until the operation
settles.
- `connection-request.ts` is the operation store: keyed by `op_id`, one entry per session,
`applyOperationStatus` / `applyConnectionUpdate` as pure reducers, `respond` leaves the
entry in place (the backend answers with `connection.update`), `ConnectionTargetOutcome`
is a discriminated union the backend's transition table accepts.
- `input-requests.ts` applies `connection.update`; `connection.expire` and the resume
snapshot correlate by `op_id` (a snapshot has no `request_id`).
- `ConnectorOffer` renders one `ConnectorCard` per target from a single
`Record<ConnectionTargetState, phase>` table; Connect opens the stored link, Try again on
failed / expired reissues through `connectors.connect` on the open operation, Not now is a
per-target `skipped`, Continue settles. A settled operation renders `ConnectorSummary` rows
with no live control.
- `tool-render-class.ts`: `manage_connections` renders the card regardless of
`HERMES_GUEST_ONBOARDING`; the flag still gates the onboarding flow, not the card. The
backend gate already decided admission; a card only exists because the tool was admitted.
- `mcp-setup-tool.tsx` speaks the same outcome vocabulary (connected / skipped / failed).
- `ConnectorRow.connectionStatus` is the seven-state literal union, not `string | null`.
- The guided-onboarding poller (`first-build-connectors.ts`) keeps its own row/phase types
and compiles unchanged; PR3 moves it onto the operation.
anti-slop: no net-new findings (17 touched files vs 11d1a12472).
* fix(connectors): the card never parks the tool thread; every update carries the snapshot
Found by the pre-PR adversarial review and a real-path E2E test (both left in the tree).
- The desktop `connection_callback` was still `_block("connection.request", ...)`, which parked
the tool thread on a private request-id Event until a `_respond` that no longer exists for
this event. `connection.respond` settled the operation but the tool waited its full deadline
before the watcher loop even started. The callback now only emits the card; the operation's
own wake loop is the wait. The MCP leg's blocking bridge goes with it: the card answers
through `connection.respond` like every other card.
- `connection.request` and every `connection.update` frame carry the full target snapshot
(state, link, detail). The initial mint happened before the card existed, so the renderer
never saw the links and Connect stayed disabled; a Continue settlement stamped
`not_connected` on the backend while the card still showed `initiated`. The store now
overlays the snapshot; no state is reconstructed from deltas.
- The `connection.update` emitter is a class-level `on_change` slot on the operation, set
once by `register()` (a second `register()` no longer stacks wrappers); session lookup takes
`_sessions_lock`; a re-minted link on an `initiated` target goes through `refresh_link()`
and emits, instead of a bare attribute write.
- `session.interrupt` is checked before the first observe, so an interrupted call settles
`interrupt`, not `all_resolved`.
- A gateway list reporting `expired` for an initiated target is recorded with actor `clock`
(the contract's owner of that edge); it raised `IllegalTransition` before.
- Dead `keepWaiting` i18n keys from the deleted renderer poller removed.
tests/tui_gateway/test_connector_operation_e2e.py runs the desktop lifecycle through the real
tool, registry, gateway RPC handlers and callback bridge with only the HTTP client faked.
* docs(connectors): prompts and docs describe the operation, not the deleted wait verb
The onboarding prompts told the model to call action="wait" with timeout_seconds and to
expect a hidden [setup]/[connectors] note; both are gone. tool-search.md and
toolsets-reference.md said the model gets a connect link on the desktop. tui_gateway/AGENTS.md
gains the connection-operation row of the surface table.
* fix(connectors): the panel re-mints only a dead link
Try again on a failed or expired target mints a fresh link on the open operation. A waiting
target keeps the link it was minted with; the card reopens it and connectors.connect refuses
to spend a second mint (LINK_STILL_VALID). The unused refresh_link() goes. The package
docstring names the new siblings; the nine-name public surface is unchanged.
* test(connectors): the local-batch test answers the operation the way the card does
The callback stopped returning an answer in f782b26d98 (the card answers through
connection.respond); this test still returned one and waited out the 300s deadline in CI.
* ci: retrigger
* fix(connectors): the desktop card appears outside guided onboarding
Live on a signed-in macOS desktop, the two-app connect never showed a card. Three
defects, each hidden by a test that bound state the running app never binds.
The backend read the surface from HERMES_SESSION_PLATFORM only. The desktop and TUI
gateway bind it as HERMES_SESSION_SOURCE (_set_session_context), so session_platform()
was "" and managed connects took the off-desktop branch: links in the model's message,
no operation. session_platform() now reads platform, then source. The E2E test binds
through server._set_session_context instead of set_session_vars(platform="desktop").
The renderer routed manage_connections to the card only under isOnboardingEnabled(),
the HERMES_GUEST_ONBOARDING launch flag, in message-parts.tsx and the run splitter in
fallback.tsx. tool-render-class.ts had already dropped that gate in this PR; the two
routers had not. Both now route on the tool name alone.
ConnectorTool resolved the session owner by the runtime id. Owner routes, hints and
session rows are keyed by the stored id, so in registry topology the owner never
resolved and the card rendered null while the tool blocked. It now resolves by the
stored id, matching the PR1.5 card and every other owner lookup.
message-parts-connectors.test.tsx mounts the real Fallback router with the onboarding
flag off and distinct runtime/stored ids; red before each renderer fix, green after.
* style(connectors): shorter comments, no module mock in the card router test
The router test mocked isOnboardingEnabled to false; jsdom has no preload bridge, so the
real function already returns false. Comments that restated the code are cut to one line.
anti-slop: no net-new findings (25 touched files)
* fix(connectors): Connect on a waiting row opens the stored link
ConnectorCard derived the button's loading state from the phase label, so a managed row that
read "Finish connecting in your browser" (every row, since links are minted up front) had a
disabled Connect button. Nothing on the desktop could open the sign-in link; every managed
connect ended skipped, not_connected, or at the deadline.
The card now takes `busy` for "the action itself is running" and keeps `phase` as a label.
The MCP card passes its in-flight flag; the connector card passes the re-mint wait. Red before:
the Connect button on an initiated row rendered disabled and a click opened nothing.
* fix(connectors): a settled card stays dead; the card binds to its tool call only
A second connect for the same apps revived the finished card on the old tool row. The
connection.request payload carried no id, so the renderer fell back to matching rows by
connector names, and any row with those names qualified, settled or not.
The operation now records the model's tool_call_id and sends it in connection.request and in
the resume snapshot. The card binds to the tool row with that id and to nothing else; the
name-match fallback is deleted. A payload without the id is rejected by the store.
`reason` is removed from the tool: it was the only text the card ever showed from the model
and its absence forked a second tool part, since `reason` doubled as the row-correlation key
in tool-parts.ts. The card never needed it.
`connection.expire` is deleted from the contract and from _EXPIRING_REQUESTS: the card is
raised with _emit, not _block, so nothing has emitted it since the operation lifecycle landed.
Sid's rule of record: a resolved card is fully dead; no path brings it back.
* fix(connectors): the watch loop settles once, on time, and never raises into the result
Three findings from the live review, one loop.
Continue racing a finished sign-in: the loop ran the gateway read, then settled. A read that
returned `connected` for an already-settled or failed target raised IllegalTransition out of
the tool and the model got a generic error instead of the per-app outcomes. The read now skips
targets that are not live (pending, initiated) and skips a settled operation; the loop checks
`settled` after every read.
Settle reason as row text: `settle()` wrote `continue`/`deadline` into each unresolved target's
`detail`, and the card printed it in red. The reason stays on the operation only.
Stop and the deadline waited for the next tick: `/stop` sets a per-thread flag with no wake
hook, so the sleep is sliced at 250 ms and the flag and clock are read each slice. The clock is
also checked before each read, not only after.
Tests: a failed mint that later reads connected settles cleanly; Continue during a read keeps
the settled result; no reason in detail; an interrupt settles within the same second.
* fix(connectors): MCP setup off the desktop returns unavailable instead of blocking
run_mcp_operation treated a non-None connection_callback as "a card exists". Every tui_gateway
session has that callback, the Ink TUI included, so an MCP install from the terminal UI blocked
until the 300 s deadline while the docs promised `unavailable` with the terminal commands.
The MCP path now reads the session surface the same way the managed path does; the callback is
never the predicate. Test binds the surface to `tui` with the callback attached.
* fix(connectors): a failed Try again shows the failure, not the old dead link
The panel's re-mint ignored the gateway's per-app status and moved the row to `initiated` with
whatever link came back, `None` included, so a mint that failed again rendered as waiting on the
link that had already died.
One reader of a mint response now serves both the first mint and Try again
(`managed.mint`, with the actor as a parameter). A repeated failure keeps the row `failed`,
drops the link, and carries the vendor's new text through `operation.refresh`, which emits a
frame without a state change so the card redraws.
* fix(connectors): a forced reconnect waits for the new sign-in before it reports connected
`reconnect` with `force: true` is the account switch. The vendor keeps the old account active
while the new link waits, so the first list read after the mint said `connected` and the
operation settled at once: the new link was dropped and the model was told the switch was done.
A forced target is marked awaiting_new_attempt after the mint. The watcher ignores its row until
the list shows the new attempt (`connectionStatus: initiated`) once, then trusts `connected`.
* fix(connectors): the operation registers under the gateway session key
The tool registered the operation under the agent's session_id; every RPC (connection.respond,
connectors.operation.status, the panel's connectors.connect) and the update emitter looked it up
by the gateway's session key. Those agree until compaction rotates the agent id mid-turn; then
the card's clicks find nothing, no update reaches it, and the tool waits out the deadline.
The registration key is now the bound HERMES_SESSION_KEY, with the agent id as the fallback for
callers with no gateway (unit tests, a bare CLI). The E2E passes a rotated agent id and drives
the card by the gateway key.
* fix(connectors): the forced-reconnect gate reads any non-active row; a failed re-mint of an expired row is failed
Three follow-ups from the verification of the fix pass.
The awaiting_new_attempt gate cleared only on the literal `connectionStatus: initiated`. The
field is optional on the wire and `initializing`, `failed`, `expired` are valid values, so a
forced reconnect could wait the full 300 s and swallow a failed new attempt. The gate now holds
only while the row still reads as the old account (`connected` or `active`) and releases on
anything else.
Try again on an `expired` row whose re-mint fails raised IllegalTransition (no expired → failed
edge). The re-mint steps through `initiated` as the user's attempt, then `failed`, then drops the
dead link.
`detail` never carries a state name any more: `failed` as detail rendered as the row label and
made agent/display.py tag the settled result as a tool error. Only vendor text goes there.
`connection.expire` removed from the renderer's unscoped-stream set; nothing emits it.
278 lines
15 KiB
Markdown
278 lines
15 KiB
Markdown
---
|
||
title: Tool Search
|
||
sidebar_position: 95
|
||
---
|
||
|
||
# Tool Search
|
||
|
||
When you have many MCP servers or non-core plugin tools attached to a
|
||
session, their JSON schemas can consume a substantial fraction of the
|
||
context window on every turn — even when only a few of them are relevant
|
||
to what the user actually asked for.
|
||
|
||
**Tool Search** is Hermes' opt-in progressive-disclosure layer for that
|
||
problem. When activated, MCP and plugin tools are replaced in the
|
||
model-visible tools array by three bridge tools, and the model loads each
|
||
specific tool's schema on demand.
|
||
|
||
:::info Built-in Hermes tools never defer
|
||
The tools that make up Hermes' core capability set (`terminal`,
|
||
`read_file`, `write_file`, `patch`, `search_files`, `todo`, `memory`,
|
||
`browser_*`, `web_search`, `web_extract`, `clarify`, `execute_code`,
|
||
`delegate_task`, `session_search`, and the rest of
|
||
`_HERMES_CORE_TOOLS`) are *always* loaded directly. Only MCP tools and
|
||
non-core plugin tools are eligible for deferral.
|
||
:::
|
||
|
||
## How it works
|
||
|
||
When Tool Search activates for a turn, the model sees three new tools in
|
||
place of the deferred ones:
|
||
|
||
```
|
||
tool_search(queries, limit?) search the deferred-tool catalog (one or more queries)
|
||
tool_describe(names) load the full schemas for one or more tools
|
||
tool_call(calls) invoke deferred tools; `calls` is an array of {name, arguments}
|
||
```
|
||
|
||
`calls` takes one entry per invocation; a single local call is an array of
|
||
one. Only `connectors__` names may be batched together; mixed and
|
||
multi-local batches are rejected.
|
||
|
||
A typical interaction looks like:
|
||
|
||
```
|
||
Model: tool_search(["create a github issue", "send a slack message"])
|
||
→ { results: [ { query: "create a github issue",
|
||
matches: ["mcp_github_create_issue", ...] },
|
||
{ query: "send a slack message",
|
||
matches: ["mcp_slack_post_message", ...] } ],
|
||
tools: { mcp_github_create_issue: { description: "...",
|
||
required: ["title"], ... },
|
||
mcp_slack_post_message: { ... } } }
|
||
Model: tool_describe(["mcp_github_create_issue", "mcp_slack_post_message"])
|
||
→ { tools: { mcp_github_create_issue: { parameters: { ... } },
|
||
mcp_slack_post_message: { parameters: { ... } } } }
|
||
Model: tool_call({ calls: [{ name: "mcp_github_create_issue",
|
||
arguments: { title: "...", body: "..." } }] })
|
||
→ { ok: true, issue_number: 42 }
|
||
```
|
||
|
||
Each query in a `tool_search` call is searched independently against the
|
||
same catalog (`limit` applies per query); the per-query groups carry tool
|
||
names only, while the shared `tools` map holds each matched tool's
|
||
description and required parameter names once. Queries are stemmed, so
|
||
"issues" finds `create_issue`. Each query group that returns no matches
|
||
includes an `available_sources` summary of the connected servers so a lexical
|
||
miss is not mistaken for a missing capability.
|
||
`tool_describe` resolves every requested name in one call; unknown names
|
||
are reported in `not_found` without failing the rest of the batch.
|
||
|
||
When the model invokes `tool_call`, Hermes **unwraps the bridge** and
|
||
dispatches the underlying tool exactly as if the model had called it
|
||
directly. Pre-tool-call hooks, guardrails, approval prompts, and
|
||
post-tool-call hooks all run against the real tool name — not against
|
||
`tool_call`. The activity feed in the CLI and gateway also unwraps so you
|
||
see the underlying tool, not the bridge.
|
||
|
||
## When does it activate?
|
||
|
||
Tool Search uses **tiered disclosure**: the presence of *any* deferrable
|
||
(MCP/plugin) tool activates the bridge; what scales with catalog size is
|
||
how much of the catalog stays visible, not whether schemas defer.
|
||
|
||
| Tier | Condition | What the model sees |
|
||
| --- | --- | --- |
|
||
| **0** | No MCP/plugin tools | Every tool eager, no bridge. Pass-through. |
|
||
| **1** | Deferred catalog's listing fits the budget | Bridge + a skills-style manifest of every deferred tool (name + short description, degrading to names-only when over budget). Degradation is **per server**: when one oversized server (Cloudflare) is attached alongside small ones (Linear), the small servers keep their per-tool listings and only the oversized server collapses to a summary line. |
|
||
| **2** | Per-tool listing exceeds the budget even names-only for every server (e.g. Cloudflare's flat API surface alone: ~3,300 tools whose names are ~32K tokens) | Bare bridge + a one-line-per-server summary (server name + tool count), so the model knows which domains are reachable; individual tools are discoverable only through `tool_search`. |
|
||
|
||
The listing budget is `min(threshold_pct% of context, listing_max_tokens)`.
|
||
The decision is re-evaluated every time the tools array is built, so
|
||
adding or removing MCP servers mid-session moves the session between
|
||
tiers on the next assembly.
|
||
|
||
## Configuration
|
||
|
||
```yaml
|
||
tools:
|
||
tool_search:
|
||
enabled: auto # auto (default), on, or off
|
||
threshold_pct: 5 # listing budget as a percentage of context
|
||
search_default_limit: 5
|
||
max_search_limit: 25
|
||
listing: auto # embed a grouped name+description catalog manifest
|
||
listing_max_tokens: 4000
|
||
```
|
||
|
||
| Key | Default | Meaning |
|
||
| --- | --- | --- |
|
||
| `enabled` | `auto` | `auto`/`on` activate whenever at least one deferrable tool exists; `off` disables entirely (everything stays eager). `auto` is currently an alias of `on` — it is reserved for a future mode that inlines schemas when they fit the context and defers only when they don't. Pin `on` or `off` if you want today's behavior guaranteed across upgrades. |
|
||
| `threshold_pct` | `5` | Listing budget as a percentage of the active model's context length. Range 0–100. |
|
||
| `search_default_limit` | `5` | Hits returned per query when the model calls `tool_search` without a `limit`. |
|
||
| `max_search_limit` | `25` | Hard upper bound the model can request via `limit` (per query). Range 1–50. |
|
||
| `listing` | `auto` | Embed a skills-style manifest of every deferred tool (name + first sentence of its description, ≤60 chars, grouped by MCP server) in the `tool_search` bridge description. `auto` includes it when it fits the budget (falling back to names-only, then to the tier-2 server summary); `on`/`off` force either way. |
|
||
| `listing_max_tokens` | `4000` | Absolute cap on the embedded listing, regardless of context size. Range 200–60000. Large catalogs degrade to names-only or per-server summaries, keeping full schemas available through search. |
|
||
|
||
Per-call array caps are internal safety bounds, not configuration. Over-cap
|
||
calls return an error so the model can retry with a smaller batch.
|
||
|
||
### Why the listing exists
|
||
|
||
Without it, deferred capabilities are *invisible* — live benchmarking showed
|
||
models substituting visible core tools (running `gh` in the terminal instead
|
||
of searching for the deferred GitHub tool) or declaring a capability
|
||
nonexistent instead of calling `tool_search`. The listing applies the skills
|
||
pattern to tools: every capability stays discoverable by name at all times,
|
||
while full parameter schemas remain deferred. If the model sees the exact
|
||
tool name in the listing, it can skip `tool_search` and go straight to
|
||
`tool_describe`, saving a round trip.
|
||
|
||
You can also flip the legacy boolean shape:
|
||
|
||
```yaml
|
||
tools:
|
||
tool_search: true # equivalent to {enabled: auto}
|
||
```
|
||
|
||
## Connectors (remote tools)
|
||
|
||
When you are signed in to the Nous Portal, the bridge additionally reaches
|
||
**connectors** — remote tools served by the managed tool gateway. They are
|
||
never registered locally: `tool_search` sends each query to the gateway, adds
|
||
the gateway's hits to the local catalog as documents (tagged
|
||
`source: "connectors"`, named `connectors__<connector>__<tool>`), and ranks
|
||
both with the same BM25 pass and the same rarest-token rule, so `limit`
|
||
caps the group as a whole and a connector tool that answers the query is
|
||
never pushed out by local tools that share one word with it. The gateway
|
||
call is bounded at 30 seconds; a slow or dark gateway degrades to local
|
||
results only. `tool_describe` fetches connector schemas from the gateway,
|
||
and `tool_call` sends each connector entry in a batch as its own gateway
|
||
request, in input order (a tool name the gateway does not know under its
|
||
conventional slug is retried once under the literal slug, so an entry can
|
||
cost two requests). If a connector ever shipped both `GMAIL_X` and a literal
|
||
`X`, both would compose to `connectors__gmail__X`, which runs `GMAIL_X`;
|
||
search keeps that twin, drops the other, and logs a warning. Results splice back into the batch's original order
|
||
with recomputed counts.
|
||
|
||
```yaml
|
||
tools:
|
||
connectors:
|
||
enabled: true # false — never touch connector routes; the bridge
|
||
# behaves exactly as if the feature didn't exist
|
||
```
|
||
|
||
Signed out (or when the gateway does not serve connectors for your
|
||
account), everything above is invisible: local search behaves exactly as
|
||
described in the rest of this page, with no errors shown to the model.
|
||
|
||
A connector call that needs an account you haven't linked returns a
|
||
`CONNECTION_REQUIRED` error. The `manage_connections` tool lists connectors and
|
||
their connection state and starts an authorization: in the desktop app the call
|
||
shows a card, blocks until each app is connected or skipped, and reports the
|
||
outcomes; elsewhere it returns a connect link per app for the user to open.
|
||
Disconnecting an account is done by the user in the Portal. The same tool also installs, enables and authorizes
|
||
local MCP servers from the catalog (targets with `mcp: true`), so it is
|
||
present whether or not you are signed in; only the managed-connector actions
|
||
need the sign-in.
|
||
|
||
`tool_call` accepts a batch: `calls` is an array of `{name, arguments}`
|
||
entries (a single call is an array of one). Each connector entry in a batch
|
||
is dispatched as its own gateway request, one after another; local deferred
|
||
tools stay one entry per `tool_call`. Approvals settle per entry before
|
||
dispatch, and a `/stop` between entries leaves the unstarted ones unsent
|
||
(their slots report `INTERRUPTED`).
|
||
|
||
## When NOT to use it
|
||
|
||
Tool Search trades a fixed per-turn token cost (the three bridge tool
|
||
schemas plus the catalog listing) and at least one extra round trip on
|
||
cold tools (describe → call) for the savings on the deferred schemas.
|
||
At tier 1 the listing keeps every capability visible, so the discovery
|
||
round trip usually disappears — the model goes straight to
|
||
`tool_describe`. Live benchmarking showed the listing mode matching
|
||
eager loading's task success while costing less than the bare bridge.
|
||
|
||
If you want the old always-eager behavior for a small toolset, set
|
||
`enabled: off`.
|
||
|
||
## Trade-offs that don't go away
|
||
|
||
These come from the prompt-cache integrity invariant — they are inherent
|
||
to any progressive-disclosure design, not specific to this implementation:
|
||
|
||
- **One extra round trip on cold tools.** The first time the model needs
|
||
a deferred tool, it spends one or two extra model calls to find and
|
||
load the schema. The token savings on the static side are real, but a
|
||
portion is paid back at runtime.
|
||
- **No cache benefit on deferred schemas.** A loaded `tool_describe`
|
||
result enters the conversation history (so it does get cached on
|
||
subsequent turns) but it never benefits from the system-prompt cache
|
||
prefix.
|
||
- **No provider-native validation for deferred schemas.** `tool_describe`
|
||
lets the model read a deferred tool's schema, but the provider still sees
|
||
only the generic `tool_call.arguments` object. Hermes therefore coerces and
|
||
validates the underlying arguments locally before dispatch; the concrete
|
||
tool or MCP server remains responsible for schemas Hermes cannot safely
|
||
validate, such as malformed schemas or external references.
|
||
- **Model-quality dependence.** Tool Search assumes the model can write a
|
||
reasonable search query for the tool it wants. Smaller models do this
|
||
less well; the published Anthropic numbers (49% → 74% on Opus 4 with
|
||
vs. without tool search) show the upside but also that ~26 points of
|
||
accuracy is still retrieval failure.
|
||
- **Toolset edits invalidate cache.** Adding or removing a tool mid-
|
||
session changes the bridge tools' descriptions (which include the
|
||
count of deferred tools) and the catalog, so the prompt cache is
|
||
invalidated. This is the same trade-off as any toolset edit.
|
||
|
||
## Implementation details
|
||
|
||
- **Retrieval:** BM25 over tokenized tool name, source name (the MCP
|
||
server or plugin toolset the tool belongs to, so searching `"linear"`
|
||
finds that server's tools even when a tool's own name doesn't carry
|
||
the service), description, and parameter names, with Snowball
|
||
stemming (English) applied to both the index and the query so
|
||
morphological variants match ("issues" finds `create_issue`). A tool is
|
||
a result only if it contains the query's rarest token (the one in the
|
||
fewest tool documents, so the word that names the intent: `gmail`,
|
||
`github`, `incident`, not `send` or `create`). A query whose rarest
|
||
token appears in no tool returns an empty group with the connected
|
||
sources and a retry hint, instead of `limit` tools that share one
|
||
common word.
|
||
- **Relevance floor:** a tool must match at least half of a query's
|
||
*answerable* terms (terms present anywhere in the catalog) before it
|
||
is offered — sharing one incidental word with a long query is not a
|
||
match. A hunt for a capability that doesn't exist returns no results
|
||
instead of a plausible-looking list the model rephrases against
|
||
forever. The floor only engages from four answerable terms up, so
|
||
short queries like "list issues" keep full recall, and an exact
|
||
tool-name query always matches.
|
||
- **Parallel execution unwraps the bridge.** The batch planner decides
|
||
concurrency on the *underlying* tool of a `tool_call`, not on the
|
||
literal bridge name — so an MCP server opted in via
|
||
`supports_parallel_tool_calls: true` keeps its concurrency when its
|
||
tools are called through the bridge, and `tool_search` /
|
||
`tool_describe` lookups batch concurrently like any read-only tool.
|
||
- **Catalog is stateless across turns.** It rebuilds from the current
|
||
tool-defs list every assembly — no session-keyed `Map`. This avoids
|
||
the class of bug where a stored catalog drifts out of sync with the
|
||
live tool registry.
|
||
- **The catalog is scoped to the session's toolsets.** `tool_search`,
|
||
`tool_describe`, and `tool_call` only ever see and invoke tools the
|
||
session was actually granted. A subagent, kanban worker, or gateway
|
||
session restricted to a subset of toolsets cannot use the bridge to
|
||
discover or call a tool outside that subset — the deferred catalog is
|
||
the deferrable slice of the session's own enabled/disabled toolsets,
|
||
not the whole process registry.
|
||
- **No JS sandbox.** Hermes uses the simpler "structured tools" mode
|
||
(search / describe / call as plain functions). The JS-sandbox "code
|
||
mode" some other implementations offer is a large surface area; we
|
||
skip it.
|
||
|
||
## See also
|
||
|
||
- `tools/tool_search.py` — the implementation
|
||
- `tests/tools/test_tool_search.py` — the regression suite
|
||
- The `openclaw-tool-search-report` PDF in the original implementation
|
||
PR for the research that shaped the design
|