The as_completed drain + `with` join made the parallel version WORSE than
sequential main in the common case (first endpoint succeeds fast, others
slow/unreachable): main returned at first success, the parallel version
waited for every straggler. Now: after each completion, walk endpoints in
priority order and return as soon as a success is unbeatable (all
higher-priority probes already finished); pool uses shutdown(wait=False) so
losers drain in the background. Mutation-checked: removing the early exit
makes the new timing test fail (8.2s vs <1.5s).
Rebase fold: the original PR predates ZAI_ENDPOINTS growing per-endpoint
probe_models lists; the parallel worker now preserves that candidate-model
fallback loop (was: scalar model). Tests (both mutation-checked):
- candidate-model fallback within one endpoint worker
- ZAI_ENDPOINTS priority order wins over completion order
- all-fail returns None
Z.AI has separate billing for general vs coding plans and global vs
China endpoints. On startup, detect_zai_endpoint() probes up to 4
endpoints sequentially with 8s timeout each, taking 8-9 seconds when
the first endpoints return non-200 (rate limited) before a working one
is found.
Replace the sequential loop with concurrent.futures.ThreadPoolExecutor
to probe all 4 endpoints in parallel. Results are returned in
ZAI_ENDPOINTS priority order so the preference chain is preserved.
Benchmark on macOS M4 Max, Python 3.11, Hermes v0.8.0:
Before: 8.8s (sequential: global=0.9s/429, cn=1.6s/429,
coding-global=4.3s/200, coding-cn=2.0s/200)
After: ~4.5s (single round-trip, bounded by slowest endpoint)
Signed-off-by: Merlin <merlin@merlin.me>
Follow-up on the salvaged pair: the original guard's `not msg_id` arm let an
id-less internal/synthetic event erase a tracking entry a concurrently-queued
id-bearing message's drain task still needs for recall matching (id-less
events never write entries in _dispatch_inbound_event, so they must never
pop). Tests cover: normal cleanup, id-less non-erasure, overwritten-entry
ownership handoff, TTL eviction + fresh-entry survival.
_build_msg_body_with_mentions() checks the TTL of each _member_cache
entry and returns an empty member list when the entry is stale, but
never removes the entry from the dict. Over time every group_code the
bot has ever queried accumulates a permanent entry, retaining the full
member list (potentially thousands of records per group) until
disconnect().
Fix: delete the stale entry at the point it is detected as expired.
The next call to get_group_member_list_raw() for the same group will
repopulate the cache with fresh data as before.
Symmetric with the existing TTL pattern in MessageDeduplicator, which
evicts on access.
_dispatch_inbound_event() writes session_key → msg_id/raw_text into
_processing_msg_ids and _processing_msg_texts so RecallGuardMiddleware
can find and interrupt the currently-processing message. These entries
were never removed after a message finished processing, causing both
dicts to grow unboundedly — one persistent entry per unique session key
for the lifetime of the bot.
Fix: clear both entries in the _process_message_background() finally
block, after super() returns. The guard compares the stored msg_id
against event.message_id before popping: a concurrent pending message
may have already overwritten the entry in _dispatch_inbound_event while
we were running, in which case the drain task owns it and we must not
clear it. When msg_id is absent (nothing was written at dispatch time)
the pop is a safe no-op.
Note: _msg_content_cache already bounds itself to 200 entries at the
same write site; _processing_msg_ids and _processing_msg_texts had no
such bound.
Simplify-pass fold: SQLITE_MAX_VARIABLE_NUMBER is 999 on pre-3.32\nbuilds (which the repo still supports — the trigram-availability\nmachinery exists for exactly that class), and limit=10000\nlist_sessions_rich callers exist in web_server. Chunk inside the\nbatch helper — the single choke point — so no call site can overflow.
Adds two regression tests for the #59077 batch: (1) _get_session_rich_rows_batch(compact_rows=True) uses the schema-derived compact projection (no system_prompt, git_branch/git_repo_root kept); (2) list_sessions_rich(compact_rows=True) threads compact_rows through the compression-tip projection call site. Mutation-checked: hardcoding compact_rows=False at the call site fails test 2.
list_sessions_rich()'s compression-root projection called
_get_session_rich_row() once per root — a separate single-row query per
compression root on every session-list render. Resolve every tip id
first, then fetch all tip rows in one WHERE id IN (...) query via the
new _get_session_rich_rows_batch().
_get_session_rich_row() is now a thin wrapper over the batch method, so
the enriched SELECT (preview + last_active) lives in exactly one place —
future column changes (e.g. #42196's include_system_prompt) only touch
one query.
get_compression_tip()'s chain walk is untouched; it's a genuine
per-session graph walk with branch/delegate-exclusion and race handling,
and batching it safely is out of scope here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simplify-pass fold: the four copy-pasted .replace blocks meant a\nfifth pinned statement could forget its strip line — a hard 'no such\nindex' crash on read-only DBs, the exact bug the fallback prevents.\nLoop over the attribute names instead.
The INDEXED BY pin is a hard dependency -- SQLite raises 'no such
index' when the named index is missing. That happens in production:
the web dashboard's usage analytics (_get_usage_analytics,
_get_models_analytics) open state.db read_only=True, which skips
_init_schema, so a DB last written by a pre-index version has no
idx_messages_assistant_calls_by_session and every insights call
crashes with OperationalError (reproduced E2E).
Probe sqlite_master once in __init__ and strip the pin from the four
prepared statements when absent -- identical rows, optimizer-chosen
plan, no crash. Replaces the change-detector test that froze the
crash as intended behavior with a fallback-equivalence test.
Review follow-up (#67341): on a freshly initialized state.db (before
ANALYZE has run) the source-filtered branches of _get_tool_usage /
_get_skill_usage did not select idx_messages_assistant_calls_by_session
— the optimizer drove from idx_sessions_source_id and probed each
session's messages via idx_messages_session_active, scanning non
tool-call rows. Pin the index with INDEXED BY on all four fixed-predicate
branches so the plan is deterministic for both the unfiltered and
source-filtered scopes without depending on statistics.
Safe because the index is declared in SCHEMA_SQL (created by every
read-write SessionDB._init_schema) and every InsightsEngine caller opens
a read-write SessionDB; read-only attachments (which skip schema init)
are never used for insights.
Extract the four queries into class constants and add tests: query-plan
coverage for both scopes without ANALYZE, row-level equivalence between
pinned and un-pinned forms, and an assertion that INDEXED BY fails loudly
if the index is absent.
InsightsEngine._get_tool_usage and _get_skill_usage scan messages for
role='assistant' AND tool_calls IS NOT NULL, but no index aligns with
that predicate, so SQLite scans the full messages table on a large
state.db. Add a partial index over exactly those rows.
role and tool_calls are base columns in the messages table, so the index
lives in SCHEMA_SQL (created on both fresh and existing databases via the
executescript on every open) rather than DEFERRED_INDEX_SQL.
Adds schema regression coverage (fresh + reopened DB, plan uses the index)
and an Insights regression test proving tool/skill output is identical with
and without the index present.
Fixes#67341
Review fold-ins on top of #56768 (@Skywind5487):
- session_count_ge ran its query without self._lock, unlike every
sibling counter on SessionDB (session_count, session_count_by_source).
- Document the deliberate semantics change: session_count() defaults to
archived = 0, which is both the expensive part (full index scan,
measured 543us vs 4us on 20k sessions) and wrong for the only caller
(has_any_sessions asks 'has this install ever had sessions' -- an
archived session is still a created one).
Two places were using SELECT COUNT(*) when they only needed a boolean:
- has_any_sessions() called session_count() > 1 (full table scan)
- delete_session() used SELECT COUNT(*) WHERE id=? (full matching scan)
Fix:
- Add session_count_ge(n) to SessionDB — short-circuits via
SELECT 1 FROM sessions LIMIT n, returns bool
- has_any_sessions() uses session_count_ge(2) instead of session_count() > 1
- delete_session() uses SELECT 1 ... LIMIT 1 with fetchone() is None
- Add tests for session_count_ge
The no-SDK fallback guards check '"Name" in globals()' — correct on
main where a failed module-level import leaves those names undefined,
but the deferred-import port pre-binds every SDK name to None, so the
guard was always true and the fallback paths called .builder() on None
(AttributeError) wherever lark_oapi isn't installed. Local runs passed
because lark IS installed here; CI's default env has no feishu extra.
Rewrote all 14 guards to 'is not None', which is correct under both
conditions. Verified by simulating CI with a lark-blocking meta_path
hook: 74 passed, 18 skipped (the skipUnless set), zero failures.
CI exposed the whole class: feishu tests across MANY files (thread
routing, text batching, sdk executor, ...) inject a mock _client and
skip connect(), so the deferred import leaves the request-builder
globals None. Replace the single-file setUpModule with a session-scoped
autouse conftest fixture that binds the globals once when lark_oapi is
installed; when it isn't, the affected tests already skip via their own
skipUnless guards. Full tests/gateway run: zero failures beyond main's
pre-existing baseline (sorted failure-diff).
Salvage of #57657, ported onto the plugin layout (the adapter moved
from gateway/platforms/feishu.py to plugins/platforms/feishu/adapter.py
since the PR's base). lark_oapi takes seconds to import and holds the
GIL doing it; the module-level import made every gateway boot pay that
cost even with Feishu unconfigured.
- _load_lark_oapi() with double-checked locking binds the SDK globals
on first use; connect() and _standalone_send() call it via
asyncio.to_thread so the loop never blocks on the import.
- probe_bot() also calls _load_lark_oapi() (sync context) so the SDK
probe path is preserved rather than silently degrading to the HTTP
fallback before a first connect.
- check_feishu_requirements() is install-only and no longer rebinds
globals; test_feishu.py gets a setUpModule that binds them eagerly
for tests that inject fake clients.
Includes the dedicated lazy-import test file (check-does-not-import,
connect-loads-on-worker-thread).
Re-derivation of #33612 by @LeonSGP43 onto the rewritten scheduler (the
original is 10,692 commits behind; its tick() no longer exists in that
shape, so this is a fresh minimal fix crediting the PR's idea).
The gateway's built-in ticker calls tick(verbose=False) every 60s. The
idle early-return was gated on 'verbose and not due_jobs', so idle
GATEWAY ticks fell through to load_config() + worker-pool resolution
every minute. Return early on ANY idle tick; keep the post-tick MCP
orphan sweep (main intentionally reaps orphaned stdio children on idle
ticks).
3 new tests; mutation-checked (restoring the verbose-gated guard fails
the config-skip test). 66 scheduler tests green.
Re-derivation of PR #39399 onto current main: pass --prefer-offline to
the web-UI workspace install (both silent and verbose arms of
_install_web_deps) and to the update-time Node dependency refresh in
_update_node_dependencies, so npm reuses its local cache instead of
re-fetching metadata. Test expectations updated to match, mirroring the
PR's own test-update commit.
Re-derivation of PR #65645 onto current main: _build_gateway_agent_history
already runs inside a turn whose config was loaded once into
ctx.user_config; re-reading config from disk via _load_gateway_config()
per turn is redundant. Reuse the loaded turn config.
Review follow-up on the #56798 salvage: the gate shipped fully dormant
(no provider profile sets supports_prompt_cache_key, no production
caller passes it, and no plain 'openai' profile exists to set it on) —
AGENTS.md rejects dead code wired in without E2E proof.
Activate the one endpoint where the field is first-class: exact-host
api.openai.com (OpenAI documents prompt_cache_key; GPT-5.6+ docs
recommend it for cache routing). Deliberately NOT substring matching —
Azure/OpenAI-compat endpoints may reject unknown fields and stay
opt-in via the flag. 4 new tests (imply + 3 spoof/proxy/Azure
negatives); mutation-checked (substring-weakened host check fails the
spoof tests).
The runtime footer (`/footer`) shows what model ran and how full the context
is, but not how long the turn took. On a messaging platform there is no
progress bar and no shell timer — a turn that took 4 seconds and one that took
four minutes produce visually identical replies. Users comparing models,
providers, or reasoning levels have no at-a-glance signal for the one
dimension they most often care about, and "was that slow or did I imagine it?"
is unanswerable after the fact.
Adds a `latency` field to the existing footer machinery, rendering the
wall-clock duration of the agent run: `<1s`, `22s`, `1m05s`.
`gateway/run.py` measures with `time.monotonic()` immediately around the
`self._run_agent(...)` await in `_handle_message_with_agent` — the same
function that already builds the footer, so the value is the user-perceived
turn duration (monotonic, so it is immune to wall-clock/NTP adjustment).
`latency` is deliberately NOT in `_DEFAULT_FIELDS`. It is opt-in via
`display.runtime_footer.fields`. Every existing footer — and every footer a
user has today without touching config — renders byte-identically.
This is enforced by tests, not just asserted:
- `test_latency_not_in_default_fields` pins the default tuple.
- `test_resolve_footer_config_default_fields_exclude_latency` pins what
config resolution produces for an untouched config.
- `test_default_footer_renders_byte_identically` pins five exact output
strings for default-config renders **while supplying `turn_seconds`** —
proving that even when the caller measures timing, a default-configured
footer does not show it.
- `test_default_build_footer_line_ignores_turn_seconds` asserts
`build_footer_line(...) == build_footer_line(..., turn_seconds=125.0)`
under default fields.
Adding `latency` to `_DEFAULT_FIELDS` fails 11 of these tests.
No new config surface (reuses `display.runtime_footer.fields`), no new env
vars, no new core tool, no new model-facing schema. One new module-private
helper (`_format_latency`), one new keyword argument threaded through the two
existing footer functions, and 3 lines in `gateway/run.py`.
`turn_seconds` defaults to `None` and the field is skipped when it is `None`
or negative, so any call site that does not measure timing keeps working
unchanged.
`tests/gateway/test_runtime_footer.py` (+185): `_format_latency` boundary
table (sub-second, rounding at 59.4/59.6, the `m{:02d}s` zero-pad, 60m), the
render/skip/opt-in matrix, field-order placement, `build_footer_line`
threading, and the byte-stability block above.
RED-proved by mutation — each of these breaks tests:
- `latency` added to `_DEFAULT_FIELDS` → 11 failures
- dropping the `turn_seconds is not None and >= 0` guard → 2 failures
- `{sec:02d}` → `{sec}` → 6 failures
- `build_footer_line` not threading `turn_seconds` → 1 failure
51 passed in `tests/gateway/test_runtime_footer.py`; 54 passed across the
footer blast radius. `ruff check` clean.
The existing test only mutated the miss-path return; a mutation to
'return _PROVIDER_LIST_CACHE' (aliasing the global cache) survived the
suite. One line pins the cached-return copy. Mutation-checked.
The retry loop gates on real time.time() < sleep_end; with sleep mocked
to a no-op the test hot-spun 7.5 wall-clock seconds. Advance a fake
clock by each sleep amount instead (pattern precedent:
test_session_activity_persist.py).
Empty content retries previously fired back-to-back with no delay,
wasting up to 3 rapid API calls, and could not be cancelled mid-wait.
Apply the same jittered_backoff() already used for rate-limit and
API-error retries, sleeping in small increments so a user interrupt
aborts the wait instead of blocking until it elapses.
Fixes#35230
_adapter_config_interactive() imported get_env_var and set_env_var from
hermes_cli.config, but these do not exist — the actual functions are
get_env_value and save_env_value. This caused an ImportError at runtime,
breaking the entire LINE platform adapter setup.
Pain before: Any user who ran the LINE adapter setup function would get:
ImportError: cannot import name 'get_env_var' from 'hermes_cli.config'
Fix: Import the correct functions with aliased local names:
from hermes_cli.config import get_env_value as _get_env, save_env_value as _set_env
Also fixed an indentation bug introduced during the fix: the 'if value: _set_env()'
block was incorrectly nested inside the except clause.
PR: N32 (hermes-agent audit)
Review fold on the #76341 salvage: the substring test ('gh' in
source.lower()) classified GH_TOKEN and GITHUB_TOKEN as gh_cli, so a
user's env-var-specific suppression was silently bypassed (and
suppressing gh_cli silently dropped env tokens). Pre-existing bug on
main, but the PR's early gate makes the classification decide whether
the exchange runs at all. Match resolve_copilot_token's exact
'gh auth token' sentinel instead.
Adds 3 regression tests: env-var suppression gates the exchange,
gh_cli suppression doesn't swallow env tokens, all-sources suppression
skips the resolve subprocess entirely. Also corrects the ~13s comment
(actual worst case ~35s: 3x10s timeouts + 4.5s backoff).
The all-sources suppression gate now runs before resolve_copilot_token(),
which shells out to `gh auth token` (~30ms) on every pool load. A user
who suppressed every copilot source (hermes auth remove copilot gh_cli
suppresses gh_cli + all env variants) still paid the subprocess spawn on
every load — model picker open, /model, agent startup.
Enumerate the same source space credential_sources._remove_copilot_gh
suppresses and bail before any work when all are suppressed. Measured:
model.options payload build drops from ~0.46s to ~0.26s cold for an
all-suppressed user; resolve_copilot_token() is no longer called at all.
The copilot branch of _seed_from_singletons ran the suppression gate
_after get_copilot_api_token(), which retries the network exchange 3x
with backoff (~13s worst case). A source the user already suppressed
(hermes auth remove copilot gh_cli) still burned the full exchange dead
time on every pool load — model picker open, /model, agent startup —
only to have the entry discarded afterwards.
Move the _is_suppressed() gate ahead of the network call, matching the
early-gate pattern every other singleton branch uses. Suppressed copilot
sources now skip the exchange entirely. Measured: model.options payload
build drops from ~13s to ~0.2-0.4s for a user with copilot suppressed.
Add regression test test_load_pool_skips_exchange_for_suppressed_copilot
asserting the exchange is never invoked for a suppressed source.
build_local_transcribe_kwargs read stt.local.no_speech_prob_threshold /
stt.local.logprob_threshold only for Hermes' post-filter
(_is_hallucinated_segment). faster-whisper's model.transcribe() never
received them, so its internal defaults (no_speech_threshold=0.6,
log_prob_threshold=-1.0) always applied and silently dropped
low-confidence segments before they reached the post-filter — making
those config knobs dead for the first gate.
Non-English speech decodes at a lower avg_logprob, so the English-tuned
defaults discard whole utterances (empty transcript despite correct
capture and language detection). Map the same config values through to
model.transcribe() so both gates stay in sync and the knobs work.
Defaults are unchanged, so behavior is identical unless a user tunes them.
Fixes#74178
kshitijk4poor