Muse Spark accepts images on user turns but returns HTTP 400
invalid_request_error 'messages[N].content did not match any supported
type' when the vision_analyze multimodal envelope lands in a role:tool
message. With the profile veto now honored by the vision fast-path gates,
declaring the limitation routes tool-result images through the aux-LLM
text path while user-message vision stays enabled.
Fixes#101668
Refs #47742
A ProviderProfile that declares supports_vision_tool_messages=False accepts
images in user messages but rejects list-type tool-result content with 400
(xiaomi/MiMo "text is not set"). supports_vision=True alone used to flip
_supports_media_in_tool_results to True, and a vision-capable capability
lookup could re-open _should_use_native_vision_fast_path — so the native
multimodal envelope landed in a role:tool message and 400'd every turn.
Both gates now go through one _profile_rejects_tool_media() veto.
Refs #89981
(cherry picked from commit daed88f940a6a475f12bb435498e48181da60f4d, trimmed)
The naive line scanner's open() regex matched the human-readable phrase
"next picker open (or refresh)." inside the warning string, tripping the
Windows footgun gate. Reword to "next picker open or refresh."
With the picker served from resident caches only, a cold Nous row renders
every model locked (free_tier_pending) until the background prewarm lands.
Surface why on the row's existing warning slot so the user isn't left with
an unexplained greyed-out list; never override an auth warning.
Picker opens use only process-resident pricing (cached_only) and start a
single-flight daemon prewarm keyed by (profile, endpoint scope); explicit
refresh stays synchronous. Nous fails closed (free_tier_pending) until the
entitlement is known so a free account cannot briefly select paid models.
Free-tier cache becomes per-profile.
Squash of the 5-commit PR #92253 branch (d5b2070ef8..28313ff963), applied
via diff on current main; two adjacent-insertion conflicts resolved by
keeping both sides.
The contributor fix covers remote rows. The reporter's video shows the
sibling shape: with a remote gateway active, the LOCAL twin carries the
'default-this-device' alias, and message_agent's local resolver only
knows bare profile names / 'hermes'. Emit the same target annotation
whenever a local row's alias differs from its resolvable handle.
The Bot Mode mention middleware built message_agent targets from
botHandle(), which prefers a roster row's source-qualified UI alias
("default-vera"). Neither resolver accepts that form — the relay matches
canonical handle/profile (± @connection-id) and the local path a bare
profile name or "hermes" — so remote handoffs died with "No teammate
named" before enqueue.
Annotate the canonical form instead: profile@connection-id for remote
rows, canonical bare handle (default→hermes) for local ones. Pin the
profile@connection form on the relay side too, so the emitted target
stays inside the documented resolver contract (#97678).
Review fold-in on the salvage of #101877:
- `_deliver_result` routed to the durable queue whenever the worker's
`_HERMES_CRON_EXTERNAL_WORKER` marker was set, regardless of WHICH job was
delivering. A worker whose script dispatches another job in-process
(`hermes cron run <other>`) inherits that env and would have queued the
nested job's message under the outer execution id — `INSERT OR IGNORE`
then drops it silently. Match the marker against the delivering job's own
`execution_id`, as `run_one_job` already does. Regression test added
(mutation-checked: fails with the guard removed).
- A `pending` row left queued at the worker's wait timeout was still
reported as a delivery error, so `mark_job_run` recorded
`last_status=delivery_failed` for a message the next gateway's drain goes
on to send, and nothing ever corrects the job record. Log and return
success instead; the deliveries row is the authority for the send.
- Reuse `cron.executions._TERMINAL_STATES` in the parent wait loop instead
of a second hardcoded terminal set.
Follow-up to the salvaged restart-safe worker (#101877):
- delivery_queue: a row still `pending` at the worker's wait timeout was
marked `failed` and never drained, so any gateway outage longer than the
300s budget (e.g. a restart that runs `hermes update`) silently lost the
delivery. Unclaimed rows are certainly unsent, not uncertain — leave them
queued for the next gateway; only mid-send rows are fenced `unknown`.
- delivery_queue: stop running the full-table prune UPDATE+COUNT inside
every transaction (each `get_status` poll paid for it; terminalizing
paths already prune explicitly); poll at 1s instead of 250ms.
- delivery_queue/executions: use `hermes_state.apply_wal_with_fallback`
(bare `journal_mode=WAL` raises on NFS/SMB homes) and the race-safe
`hermes_cli.sqlite_util.add_column_if_missing`; drop the copied
owner-liveness helpers in favour of the ones in cron.executions.
- scheduler: the parent waited on the worker by re-opening the executions
ledger every 50ms for the whole run (~20 opens/s, hours). Wait on the
process with a 1s timeout instead — the worker commits its terminal row
before exiting — and reap stranded payload/ack files once terminal.
- scheduler: skip the housekeeping drain until a worker has actually
created deliveries.db, so non-systemd gateways never open it.
- scheduler: set up hermes logging in the detached worker entrypoint; it
runs with stdout/stderr on DEVNULL and previously logged nowhere.
- tests: test_lost_fire_claim_stops_stale_delivery still mocked
`mark_execution_running -> None`, which now means "ownership lost, return
before run_job" — the test passed without ever reaching the path it
names. Mocking `{}` restores it (mutation-checked).
Review follow-ups on the final salvage stack:
- `fast_safe_load(f) or {}` collapsed a falsy non-mapping root (`[]`) to `{}`
before the strict isinstance check, so an empty-list config evaded the
raise the previous commit added. Only map a None document (empty file) to
`{}`; every other non-mapping root now hits the strict branch.
- Docstring: the strict flag covers non-mapping roots too, not just parse
failures.
- Tests: assert the tolerant call's return per shape instead of merely
calling it; add the empty-list-root case.
A config.yaml whose top-level value is a list or scalar parses fine, so the
strict check_config_version(raise_on_parse_error=True) from #101778 still
returned (0, latest) and migrate_config() proceeded: sanitize_env_file()
rewrote .env, then save_config()'s fail-closed guard raised RuntimeError.
Raise InvalidUserConfigError up front for that shape too, so the
"no side effect before the invalid config is surfaced" guarantee holds
for both invalid-config shapes. Tolerant callers are unchanged.
Tests: parametrize the two #101778 regression tests over malformed-yaml
and list-root; assert the tolerant call still does not raise. Also make
the test_update_autostash check_config_version mock kwarg-tolerant,
matching the author's fix in test_config.py.
Post-review cleanup on the salvage stack:
- The re-warn-after-reset guard now drives on_session_reset() instead of
the private helper, so a site regressing to a bare rearm-zero fails it.
- One _over_threshold_warnings() helper replaces four inline caplog filters.
- Comment the redundant None check that narrows current_tokens for ty.
Widens the Windows parent-death fix to the class: the kernel inherits
the read end of a pipe whose only write end lives in the host, so host
death by any means (SIGKILL, OOM, crash) is EOF and the kernel exits.
Stdin EOF alone only reaches the runner between cells, so a kernel
SIGKILLed mid-cell used to outlive its host indefinitely. The
integration test now runs on every platform and is trimmed to the
contract (kill host mid-cell, kernel gone), not the handle plumbing.
Same attached-cell guard as the local kernel host (#101861): a remote
kernel mid-cell is never reaped or cap-evicted, so a fan-out never has
its runner killed under a live poll loop.
Local session kernels sweep idle-expired entries and enforce a
process-wide cap (DEFAULT_MAX_SESSION_KERNELS) on every call
(tools/code_kernel.py's _reap_unlocked / _evict_over_cap_unlocked). The
new remote kernel host (#96991) never got the same treatment:
_REMOTE_KERNELS only shrinks lazily when a specific key is revisited and
found dead, so an owner that opens kernels for several distinct
(env_type, task_env_id) combinations (or delegated children) and never
revisits some of them accumulates host-side bookkeeping entries for the
life of the gateway process.
Note this is narrower than the local case: the remote runner already
self-reaps on its own idle timeout, and SSH/Docker connections are
independently bounded by their own transport-level lifecycles (SSH
ControlPersist, Docker's session-scoped idle-timeout in terminal_tool.py)
— so nothing here leaks a live remote connection. What's missing is
purely the host-side dict/cap bookkeeping symmetry with local kernels.
Adds _reap_unlocked/_evict_over_cap_unlocked mirroring the local
implementation, reusing the same max_session_kernels config as an
independent cap on _REMOTE_KERNELS.
Follow-up to the salvaged #101894 (@jwilson411). The over-threshold
"reclamation did not run" warning is deduped on (reason, rearm mark), and
the key was only cleared when a prune committed. Every other path that
zeroes the rearm mark — compress(), on_session_reset/on_session_end,
bind_session_state, update_model — left the key in place, so a lockout
that warned at rearm=0, then a full compaction, then the same lockout
again was silent, contradicting the helper's own "warns again" contract
(and leaking the key across sessions on a rebound compressor).
- ContextCompressor._reset_proactive_prune_rearm(): one helper for the
five rearm-to-zero sites; clears the dedup key alongside the mark.
- _warn_reclamation_no_op(): dropping back under threshold releases the
key (mirrors _clear_context_overflow_warn semantics on the agent side).
- test_proactive_prune_loop_wiring: the attempts_exhausted fixture now
models the only state the real engine can produce for that branch
(should_compress() is should_compress_info()[0]) — budget spent
(max_compression_attempts=0) with the engine saying RUN, instead of
should_compress=False paired with (True, None).
- Two guards: lockout warns again after a rearm reset; dropping under
threshold releases the key. Both fail with the clears removed.
Message-only rearm could sit just above the body estimate while provider
prompt_tokens (system + tool schemas) already exceeded threshold_tokens,
so prune no-oped forever with no log. Bypass that rearm short-circuit on
the billed basis, warn once when over-threshold reclamation no-ops, and
name attempts_exhausted when should_compress_info says run but the loop
skips.
Fixes#101889
Switching to an already-open session remounted the incoming transcript and
re-tokenized every fenced code block from scratch on the main thread (N blocks
x full shiki tokenization per switch, 96-100% CPU for seconds).
- shiki-block: content-keyed LRU cache of highlighted HTML (theme scope +
language + code); remounts of unchanged blocks paint cached markup with
zero highlighter calls; misses debounced, failures degrade to plain text
- use-session-actions: warm resume keeps the session-slice array when the
reconciled content is equivalent (same guard as the cold path), so the
runtime repository and every row keep identity
- transcript-window: per-session sticky window memos; a warm re-visit with an
unchanged transcript reuses the windowed slice by reference (no re-index,
no repository rebuild); sticky cut survives switches for sessions that grew
- perf regression guards: remount must not re-tokenize (codeToHtml called
once per unique block), windowed slice reference preserved across switches,
messageComponents identity stable across session switches
Composition of #92581 on #100985: the hard cap is a constructor constant,
so raising the pool max in Settings would have left new spawns queued
behind the launch-time value. Add setLimit(); slot hand-off now goes
through a single #drain that respects the current cap, which also fixes
the original release path handing a slot to the next waiter even when the
cap had just been lowered (test: lowering never revokes granted slots; new
requests queue until under cap). main.ts constructs from
poolLimits.maxBackends and pushes changes from setPoolLimits(); pinned by
a wiring test.
Hover-intent prewarm sweeps across the Bots rail spawned past the pool cap,
LRU-evicting the backend the user was about to click — an evict/respawn
cascade that made profile switching progressively slower (#91545).
- prewarmProfileBackend skips speculative spawns once every pool slot holds
an open socket; the real click still spawns on demand.
- Pool max/idle become a device preference (Settings -> Advanced), persisted
atomically in userData (pool-limits.json) and applied live over IPC; the
HERMES_DESKTOP_POOL_* env vars remain the initial fallback. Defaults are
unchanged (3 backends / 10 min idle).
Squash of the 3-commit PR #92581 branch (a00dc088c5..783899d12f) applied
via diff onto the spawn-coordinator salvage; import + constant-block
conflicts resolved so the coordinator is constructed from, and follows,
the live preference (setLimit added in the next commit).
Follow-up to the spawn coordinator: the queued ticket waited up to
POOL_IDLE_MS (10 min) for a free local slot, but the renderer gives up on
a backend boot after 45 s. A user clicking a 4th profile with 3 fresh
backends open would see the generic "backend didn't come up" error while
the ticket kept the pool key hostage, so every later click joined the same
stale wait. Cap the wait at 30 s, log the slot pressure when it happens, and
pin the relationship to BACKEND_BOOT_WAIT_TIMEOUT_MS with a wiring test
(fails when the timeout is reverted). Also eslint --fix on the salvaged
files (import order was a lint error).
Desktop could spawn a local hermes serve per profile with no hard cap on
starting+running children: LRU eviction spares keepalive-fresh entries, so
a roster refresh across many profiles became a process wave (40+ backends,
load 30-50 reported).
LocalBackendSpawnCoordinator: at most POOL_MAX_BACKENDS local backends may
be starting or running. Remote descriptors never take a slot. Queue tickets
are per request; a slot is released only after process exit is proven
(exitCode/signalCode). A rejected wait keeps the slot occupied. Pool entries
re-assert ownership at each await so an evicted entry cannot spawn a zombie.
Squash of PR #100985 (6017abbbc4 + merge), applied via diff onto current
main. Original commits were authored as 'Motor (Hermes AI) <ceo@xtremagency.com>';
attributed here to the PR author's GitHub identity.
The ownership file accumulates one record per profile per launch, and each
record can cost up to two identity probes (parent + backend) plus a stop.
On Windows those shell out to PowerShell, whose 5.1 cold starts are slow.
Without a bound, a large roster could stall boot for minutes while the
renderer's 45s backend-boot budget expires and the user stares at the
connecting screen.
- Add REAP_PROBE_TIMEOUT_MS (5s) for the orphan-reap path; the claim path
keeps the full 30s headroom for a freshly spawned backend's marker.
- Add reapDeadlineMs (5s default) as an overall budget for one reap sweep;
when exhausted, unprocessed records are preserved for the next launch.
- stopOwnedBackend now throws when the identity probe fails (not confirmed
gone) so the record is preserved instead of leaking the backend.
Verified: two cold starts complete in ~29s (was 5+ min); renderer connects
immediately after backend ready.
The salvaged #88217 test replaced Path.stat on the class with a lambda
taking one positional arg; pathlib.exists() passes follow_symlinks= and
pytest's own tmp_path teardown crashed with INTERNALERROR TypeError.
Delegate to the real stat for every path other than state.db.
Composition bug between the salvaged #101266 (in-place v29 startup
migration: swap the trigram view/triggers, FTS5 'rebuild') and #88217
(FTS_STORAGE_VERSION 2 drops the tool_calls column from the trigram
vtable, opt-in via optimize-storage). On an install still carrying the v1
vtable, the startup migration replaced the view with one that has no
tool_calls, then 'rebuild' failed with 'no such column: T.tool_calls' and
SessionDB.__init__ raised — reproduced by opening a real main-built DB.
Gate the in-place migration on the vtable not projecting tool_calls; such
installs are already offered optimize-storage, which recreates the vtable
from FTS_TRIGRAM_SQL (cron-filtered view included). E2E: main-built DB ->
opens on this branch, optimize_fts_storage() yields v2 columns and purges
the cron row. Test fixture now builds a real external-content vtable for
both layouts; new test mutation-checked against the missing guard.
Clear zero-row rebuild markers so empty databases can finish teardown and
stamp the new layout. Extend the migration regression through close/reopen
recovery for both empty and populated databases.
Keep structured tool_calls searchable through the standard FTS index while
removing their repetitive JSON from the trigram projection. Reuse the
existing optimize-storage rebuild path for deployed v1 layouts.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Replace the requires_wal-gated unit assertion (which never runs on WAL-reset-
vulnerable runtimes such as macOS 3.46) with an end-to-end test through
repair_state_db_schema that traces every _connect_repair_durable call against
the guard's enter/exit and fails on main's shape (a connect after guard-exit).
Runs on every platform. Docstring now names the actual hazard.
apply_wal_with_fallback keeps a store in DELETE where the linked SQLite has
the WAL-reset bug; the connection-reuse contract the test guards is unaffected
but its final journal_mode assertion is not satisfiable there.
Review follow-ups on the live-inode import:
- A `.db` member is now page-restored into the live file, but a paired
`-wal`/`-shm`/`-journal` member from an old or hand-built archive still went
through the rename publish — installing a foreign WAL beside the restored
database (and over a live sidecar's inode). Skip them; current backups never
ship them (_EXCLUDED_SUFFIXES), now shared as _SQLITE_SIDECAR_SUFFIXES.
- restore_quick_snapshot ignored _safe_restore_db's False and counted a
refused restore as success; honour it like run_import does.
- _safe_restore_db docstring described the pre-#90950 unconditional fallback.
`hermes import` published every zip member, including `state.db`, with
`_extract_member_atomically` — a rename that swaps the file's inode. Any
gateway, dashboard, or WebUI process holding the database open keeps its
descriptor on the now-unlinked inode: it goes on serving pre-import pages
and writing sessions no other process can see, while the sidecar WAL left
beside the new file describes the database that was just unlinked. Nothing
raises, so the import prints "Import complete" and the sessions are simply
absent from the database everyone opens next.
The live-safe path already exists: `/snapshot restore` has routed `.db`
files through `_safe_restore_db()` since #65942, writing snapshot pages
into the existing file so every open connection converges. `hermes import`
— the disaster-recovery path, reached by users who already lost something
once — never got that treatment.
Route `.db` members through it. A target that does not exist yet has no
holders and no inode worth preserving, so it keeps the ordinary atomic
publish. A refused or failed live-safe restore now raises, so the import
reports a skipped file instead of counting a silent success, and the
existing database is left untouched.
Importing an older backup over newer work stays allowed but no longer
silent: the summary reports the session/message counts the import replaced,
the same before/after evidence `restore_cron_jobs_if_emptied` uses for
`cron/jobs.json`.
Closes#100960
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ShTVU941HYygvypY9JMYE
(cherry picked from commit 8ff260a312341cb85bdf7cdd570de5ff888c6013)
Completes the constant + samples + noise-regex trio teknium named as the bar
for this heartbeat on #98371; the Telegram noise-filter and TUI retag
parametrised suites now iterate the heartbeat wording too.
The seed put the retry deadline 900s in the future, so on unguarded code the
method short-circuited on the backoff check and the protection assertions
passed anyway; only the field-reset assertions failed. Seed the deadline in the
past so a rebuild is due, and condense the guard comment.
retry_deferred_fts_recovery() is called unconditionally on every
housekeeping tick for the life of a long-running gateway process
(#100108). It checks _fts_stale, read_only, and _conn is None, but never
_db_corrupt (bcc2e65818, #101095/#101224): a handle that observed
structural corruption is supposed to stop being touched entirely (see
_try_wal_checkpoint's identical guard, and close()'s skip of the
checkpoint), but this method has no such check.
If a handle both has a deferred stale-FTS breadcrumb AND later trips
quarantine (both are plausible on the same corrupted file — the field
incidents motivating the quarantine feature describe corruption touching
FTS shadow tables and canonical btrees together), every subsequent
housekeeping tick runs a real FTS rebuild (DROP TABLE / CREATE VIRTUAL
TABLE / bulk INSERT) against the file the code has explicitly decided to
stop touching — exactly what quarantine exists to prevent.
Fixed by returning False immediately when _db_corrupt is set, mirroring
_try_wal_checkpoint's "quarantined: never touch a damaged image" guard.
The method's own contract ("never raises") is preserved — no
StateDbCorruptError is raised here, this is a quiet skip like the other
corrupt-aware call sites.
Also resets the backoff bookkeeping (_fts_stale_retry_after,
_fts_stale_retry_interval) in the same early-return, mirroring the
success path's own reset a few lines down (review feedback from
Baophan00 on the PR). Verified empirically before making this change:
_db_corrupt is set to False nowhere in the codebase outside __init__, and
the shared registry's file-replace path always constructs a genuinely new
SessionDB instance rather than clearing the flag on a live one — so no
code path today revives a quarantined handle in place, and leaving the
backoff fields untouched is inert in practice. The reset is still cheap,
harmless, and closes a real footgun for whoever adds an un-quarantine
path later: without it, a handle quarantined mid-backoff would carry a
doubled multi-minute interval into any future retry instead of starting
from the default.
Added a regression test that marks a handle stale, forces the open-time
recovery to defer via a real held rebuild lock (so _fts_stale survives
construction), sets _db_corrupt plus a pre-existing multi-minute backoff,
and asserts the retry is a no-op with both backoff fields reset to 0.0.
Mutation-verified: reverting hermes_state_schema.py makes the retry
actually run the rebuild and return True, and separately makes the
backoff-reset assertions fail with the stale pre-quarantine values still
in place.
(cherry picked from commit 3445da1d98d84bb60cb3799ef59e1fa4c619100f)