The salvaged hardening tests matched main.ts source text to assert that
fetchConnectionStatus reaches for a bearer and that Apply preflights before
persisting. A rename breaks them while a real auth regression that keeps the
substrings passes.
Make the preflight a first-class option on applyConnectionConfigAtomically so
its ordering is observable, and assert it through the seam: preflight runs
before either write, and a rejected preflight leaves both stores and the
activation untouched.
tests/stress/ was dead weight: its own conftest set
collect_ignore_glob = ["*.py"], so pytest has never collected a single
file from it, the advertised --run-stress flag was a permanent no-op,
and no CI workflow ever invoked the scripts (#93135). Rather than wire
a nightly lane for scripts that were never verified end-to-end, remove
the suite. Kanban concurrency behavior remains covered by the regular
tests under tests/hermes_cli/ and tests/gateway/.
Closes#93135.
Salvage follow-up for #88993: the preflight was defined but never
called from the prerequisites stage or the full-install path, so the
Fedora node-gyp failure (#93063) would still occur. Wire it in after
check_node in both sequences.
npm install inside install_node_deps() builds native addons (e.g. node-pty) via node-gyp, which needs a C/C++ compiler. That was never checked, so a missing g++ only surfaced as a generic "npm install failed or timed out" deep inside npm's own output — and because install_node_deps failing short-circuits the rest of main() via `|| return`, users end up with no `hermes` command and no clue why.
Profile-spawned workers (kanban bots, cron jobs) can inherit a PATH of
only version-manager dirs — observed in the wild as one nvm node dir
repeated 7x. The uv-installed browser-use binary is a POSIX sh
trampoline that resolves dirname/realpath through PATH, so it died
with 'realpath: not found … exec: /python: not found' (exit 127)
before its own Python ever started.
_base_subprocess_env now floors the child PATH via browser_tool's
_merge_browser_path (the agent-browser backend already guards the same
hazard), degrading to appending FHS bin dirs if that import is ever
unavailable. Windows is a no-op (.cmd shims don't trampoline).
Verified: unit tests + real uvx browser-use --version under a
nvm-only-PATH worker env, rc 127 -> rc 0.
get_due_jobs() fires purely off the stored next_run_at <= now, with no
check that the stored instant is still an occurrence of the schedule's
current expression. A direct jobs.json edit that narrows schedule.expr
(e.g. daily "0 7 * * *" -> weekdays "0 7 * * 1-5") keeps the stored
next_run_at computed under the old expression, so the job fires on days
the new expression excludes. The within-grace fire and the catch-up
"run once now" path both inherit the wrong instant.
Add a best-effort stale-schedule guard on the fire path: when the stored
next_run_at is not an occurrence of the current cron expression,
re-anchor it via compute_next_run() from the current expression and skip
the fire. Non-cron kinds, missing expr, croniter unavailability, and
malformed input all report a match so the fire path keeps its existing
semantics. Recomputation uses the current expression, so the re-anchor
converges and cannot defer a valid job forever.
Fixes#93049
When the managed openai-audio gateway is unavailable,
_resolve_openai_audio_client_config() raises a ValueError that names the
blocker (and, for managed-Nous users, the `hermes tools` remediation).
The boolean probe in _get_provider's explicit-openai branch flattened
that into False, so the log claimed "no API key available" and the
transcription result returned the all-provider install hint -- pointing
operators at unrelated setup instead of their managed route (#93045).
Resolve the config directly in the branch so the warning names the real
blocker, and let the dispatch's "none" fallback surface the
selection-specific error for an explicit openai choice. No fallback is
added: an unavailable selection still resolves to "none", it just
reports why.
resolve_runtime_provider() documents target_model as the explicit model
override for mid-session switches and auxiliary slots, but the custom
provider path (_resolve_named_custom_runtime) never received it and
silently substituted the provider's configured default_model instead.
This made auxiliary slots such as auxiliary.background_review silently
run the provider's default model rather than the configured one — e.g. an
ocx-proxy slot configured for gemini-flash actually executed
cursor/claude-sonnet-5, hitting upstream rate limits.
Pass target_model through to the custom runtime resolver and prefer it
over the provider's default model in both the pooled and non-pooled
credential paths.
Mark the write-only tmux load-buffer call as resolve-on-exit so a daemonized tmux server cannot retain inherited stdio and force a false timeout after the direct child has succeeded.
Completes the call-site acceptance item from #93134 as a companion to #93148.
Co-authored-by: JoaoMarcos44 <87440198+JoaoMarcos44@users.noreply.github.com>
The timeout handler only called settle(124) when resolveOnExit was true.
In the default path the promise waits for 'close', which requires every
inherited stdio handle to close — a daemonized grandchild that kept the
pipes open meant 'close' never fired, and after the timeout SIGTERM
(which only reaches the direct child) nothing settled the promise. The
await hung forever: the clipboard path (setClipboard -> tmuxLoadBuffer
-> osc.ts spawn without resolveOnExit) leaked a pending promise whenever
a spawned tool forked a stdio-inheriting daemon (#93134).
Settle(124) unconditionally in the timeout handler. The settled-guard
makes it a no-op when the child's own 'exit'/'close' won the race, so
normal timeout behavior is unchanged; in the daemon case it becomes the
only exit and returns the same 124 the close path would have.
Also un-skips the documented-hang regression test, with a 30s daemon
sleeper so it genuinely outlives the timeout (and vitest's own 5s test
timeout — before the fix the test fails by timing out, not asserting),
plus an elapsed bound.
Adversarial-review fixes for the #93057 snapshot-compaction PR:
- Fail-closed detachment: only re-enable compression after
bind_session_state successfully severs the engine's parent binding.
A failed rebind keeps the historical compression_enabled=False
behavior and warns, instead of running compaction against a
compressor still bound to the parent's SessionDB (#38727 re-open).
- Warm-cache parity: defer both compression gates (turn-prologue
preflight + pre-API pressure check) until the fork's first provider
response, so the first request replays the full snapshot as the
intended cached read and compaction applies from the second request
on — matching the documented budget mental model.
- Tests: regression for the rebind-failure fail-closed path (red on
pre-fix code) and the existing threshold-crossing test reworked to a
two-request review asserting the warm first request + compacted
second request. 116 tests green across all touched suites; ruff
clean.
Detach the review fork's compressor from the parent SessionDB/session_id
and re-enable in-memory-only compaction for oversized snapshots, instead
of the historical compression_enabled=False guard that left the fork's
replayed transcript unbounded (350k-384k input tokens per request, 1.49M
total across one 8-request review). Add an aggregate input-token budget
(auxiliary.background_review.max_input_tokens, default 600k) so repeated
tool calls cannot recreate an unbounded transcript; the tool loop stops
before the provider call that would cross it.
Closes#93057
Reproduce the schema-btree failure where the in-place writable_schema/VACUUM ladder can reduce a 3,048-page canonical state.db to 113 pages and still return repaired=False.
Move all mutating strategies behind a complete SQLite online-backup snapshot, retain one exclusive SQLite guard from staging through transactional promotion, preserve committed WAL frames and the live inode, fail closed on environmental hazards, and add adversarial regression coverage for failed-repair preservation, post-stage writer races, interrupted copies, stale scratch, disk admission, attempt-ledger semantics, and durability routing.
Fixes#93064
Supersedes the delivery mechanics of #87409 while preserving its implementation provenance.
Co-authored-by: cervantesh <11169707+cervantesh@users.noreply.github.com>
Follow-up on top of the salvaged #93335:
- context_compressor._sanitize_tool_pairs now expands alias spellings on
the RESULT side too (tool_result_id_variants), so a composite
call|item-keyed result pairs with its split-field tool_call instead of
being dropped and its call stripped.
- The compressor's _tool_call_id_variants staticmethod and
agent_runtime_helpers' module-level _tool_call_id_variants are now thin
forwarders to agent.message_sanitization.tool_call_id_variants — one
policy owner for alias expansion, so the pre-call sanitizer, repair
pass, dedup pass, and compression sanitizer can never drift apart.
- Preserved the #91768 SDK-object tolerance in repair pass 1 (the
shared helper handles non-dict tool_calls via getattr; the salvaged
commit's isinstance-dict guard was dropped in the merge resolution).
New regression tests: composite-keyed results through
sanitize_api_messages (both directions) and _sanitize_tool_pairs, with
negative controls. Sabotage-verified: compressor test fails with raw
tool_call_id tracking.
Match Responses/Codex tool-call aliases across execution, repair, sanitization, replay, and duplicate handling so valid parallel results are not replaced by unavailable stubs.\n\nFixes #93251
A newer main-side sniff fast-path skipped any file with a NUL in its
head, short-circuiting before the magic-number check the salvaged fix
added — reintroducing the #77927 bypass. Key the fast-path on
executable magic only; NUL-bearing text falls through to the tail
logic (magic check, size-before-strip, NUL-strip, scan).
The #76762 binary check treats any NUL byte in the first chunk as "compiled
binary, nothing to scan":
if b"\x00" in data:
return None, False
"Contains a NUL" and "is a compiled binary" are different questions, and the
gap between them is a guard bypass. `bash` executes a *text* script straight
past an embedded NUL, so one pad byte disables the entire scan while the
script still runs:
#!/bin/bash
# pad<NUL>
hermes gateway restart
scan("bash padded.sh") -> False (not blocked)
bash padded.sh -> executes the lifecycle command
This shape was blocked before #76762, so the crash fix traded a loud failure
for a silent one.
Keying the check on a leading `#!` is not sufficient: a shebang-less file with
a NUL on any line but the first also executes normally. (A NUL on line 1 of a
shebang-less file is the one shape bash rejects, exit 126 — but that same file
is still executable via `. file`.)
Fix: identify binaries by MAGIC NUMBER — ELF, Mach-O (incl. byte-swapped and
universal/fat), PE/COFF, static archive, gzip, zip — with a shebang always
winning. A NUL-bearing *text* file is scanned with its NULs stripped;
stripping can only splice tokens together, never apart, so it fails closed.
File extensions are deliberately not consulted, so a suffixless shell script
is still scanned.
The size check now runs BEFORE the strip: stripping shrinks the buffer, so
checking afterwards would let an oversized file slip under the threshold and
skip the fail-closed branch. (Caught by
test_oversized_nul_bearing_text_still_fails_closed, which failed on the first
cut of this patch.)
Return values are unchanged, so this does not conflict with the in-flight
crash-class fixes to the same function.
Tests (tests/hermes_cli/test_gateway_restart_loop.py), 3 of which fail on main:
- test_nul_padded_script_is_still_scanned
- test_nul_padded_script_without_shebang_is_scanned
- test_oversized_nul_bearing_text_still_fails_closed
- test_elf_binary_is_not_scanned_as_script (#76762 stays fixed)
- test_macho_binary_is_not_scanned_as_script (incl. fat binary)
- test_clean_script_without_lifecycle_command_not_blocked
`_iter_referenced_shell_scripts` recognises the `source` builtin so a script
pulled in with `source ./restart.sh` gets scanned for lifecycle commands. The
POSIX dot operator is the same builtin, but it was not caught:
if executable_name in {".", "source"}:
`executable_name` is `Path(executable).name`, and `Path(".").name` is the
**empty string** -- pathlib normalises "." to the current directory, whose name
is "". So the set membership never matched for `.`, the sourced script was
never added to the reference walk, and its contents were never scanned.
Verified against current main:
. /tmp/restart.sh -> not blocked (script never scanned)
source /tmp/restart.sh -> blocked
bash /tmp/restart.sh -> blocked
where /tmp/restart.sh contains a `hermes gateway restart` line. Sourcing runs
the script in the current shell, so the dot spelling is not merely equivalent
to `source` -- it is the more common form in practice.
Fix compares the raw token as well as the basename:
if executable in {".", "source"} or executable_name == "source":
Keeping the `executable_name == "source"` arm preserves the existing behaviour
for a path-qualified spelling, while the raw-token test catches `.` without
relying on pathlib normalisation.
Tests (tests/hermes_cli/test_gateway_restart_loop.py):
- test_dot_operator_sourced_script_is_scanned -- the regression; fails on main
- test_source_builtin_sourced_script_is_scanned -- `source` stays blocked
- test_dot_operator_clean_script_not_blocked -- widening the check must not
false-block an innocent `. ./activate.sh`
Found while auditing the guard after #76762. Scoped deliberately to this one
defect; the NUL-padded-script bypass I found in the same audit is a separate
PR.
Live WS E2E after the #93361 merge (real web_server + tui_gateway, isolated
HERMES_HOME, 2s grace): drop socket -> re-resume stored id on a new socket
still produced a ws_orphan_reap reclaim. The lazy/unpersisted resume branch
(no state.db row yet -- every fresh Bot Chat) returned the sentinel-parked
live record without rebinding its transport or cancelling the armed reap
Timer, so the storm survived for exactly the Bot Mode sessions the cluster
targeted. The unit-covered paths (_live_session_payload, _reuse_live_response,
_claim_or_reuse_live) were all correct; this branch bypassed them.
Regression test drives the real session.resume RPC against a sentinel-parked
unpersisted record (sabotage-verified: fails without the fix). After the fix
the full live E2E passes 10/10 scenarios including a 4-cycle drop/resume storm
loop with zero reclaim broadcasts.
Follow-up to #93339: the auxiliary.review slot existed in config but was
missing from every model-picker surface, so users could only set the
review model by hand-editing config.yaml.
- hermes_cli/web_server.py: review in _AUX_TASK_SLOTS (REST allowlist,
stale-aux warning sweep)
- hermes_cli/main.py: review in _AUX_TASKS (hermes model aux picker)
- apps/desktop model-settings.tsx + all 5 i18n locales (en/ja/zh/
zh-hant/ar): review slot with label/hint
- web/src/pages/ModelsPage.tsx: review row in dashboard Models page
- tests: registry-sync test pinning review across DEFAULT_CONFIG,
_AUX_TASKS, and _AUX_TASK_SLOTS (curator pattern)
- docs: aux-task table in fallback-providers.md (en) + zh-Hans mirrors
of fallback-providers and the delegation /review section missed in
#93339
Runbook prose inside a quoted-delimiter heredoc feeding a data sink
(cat > file <<'EOF') is documentation, not a command this shell will
execute. Mask provably-inert heredoc bodies (tools/shell_heredoc's
conservative stripper, already used by terminal_tool) before scanning.
Fails open on any ambiguity: executable and unquoted-delimiter heredocs
stay scanned. Salvaged from PR #88336 by @zgqq (the heredoc half; its
Branch D boundary and dir-token halves already landed/were fixed).
execute_code lacked the lifecycle guard entirely, and Python argv-list
forms (subprocess.run([...])) separated command words with brackets and
commas the shell-shaped pattern could not see. Mirror the terminal_tool
guard in execute_code (ownership-gated per #92560) and strip argv-list
punctuation in the token-join re-scan. Salvaged from PR #68289 by
@arcimun, adapted to the ownership gate and current guard structure.
The gateway-lifecycle guards in cron/lifecycle_guard.py (Branch B, the
unconditional hard-block used by cron creation and the terminal tool when
_HERMES_GATEWAY=1) and tools/approval.py's launchctl rule both matched
`launchctl <verb> ... hermes[.-]?gateway` as a single sequential regex,
requiring the hermes-gateway label to appear literally AFTER the verb.
A shell command that builds the label earlier in the string — e.g. a
for-loop reading labels from a list defined before the actual launchctl
call — defeats that ordering entirely:
for item in 'ai.hermes.gateway-apollo:...' 'ai.hermes.gateway:...'; do
label=${item%%:*}; plist=${item#*:}
launchctl bootout "gui/$uid/$label"
launchctl bootstrap "gui/$uid" "$plist"
done
The literal text "hermes.gateway" only ever appears in the for-list,
never after "bootout" — so `[^\n]*\bhermes[.\-]?gateway` never matches at
the verb's position, even though the command unambiguously targets the
gateway's own launchd label.
cron/lifecycle_guard.py's verb list also didn't include `bootout` at all
(present in tools/approval.py's list and covered by its own test suite —
`launchctl bootout ai.hermes.gateway` is explicitly asserted as dangerous
there — so the omission in the sibling file looks like list drift between
the two guards rather than an intentional exclusion).
`bootout` is the verb that actually deregisters a launchd job (unlike
kickstart/stop, which just bounce a still-registered one), so a command
using it evades both guards, then removes the service from launchd with
no supervisor left to bring it back — worse than a simple restart-loop.
We hit this for real: a gateway self-restart (triggered from a chat
request to change the default model) used a raw terminal `launchctl
bootout`/`bootstrap` loop across 4 launchd labels instead of the normal
`hermes gateway restart` path. It slipped past both guards, self-bootout
killed the process mid-drain before its own follow-up bootstrap could
run, and all 4 gateway profiles ended up fully deregistered from launchd
with zero user approval (approvals.mode: manual was configured) until
someone manually re-bootstrapped them.
Fix: both guards now check "a launchctl lifecycle verb appears somewhere
AND a hermes-gateway label appears somewhere", independent of order, and
cron/lifecycle_guard.py's verb list gains bootout/kill/disable/remove to
match tools/approval.py's existing set. Internal recovery code
(hermes_cli/gateway.py's own `subprocess.run(["launchctl", "bootout",
...])` calls) is unaffected — these guards only scan shell-command
strings composed by the agent's terminal/cron tools, not the CLI's
trusted internal subprocess argument lists.
Adds regression tests in both test files reproducing the exact incident
command (label built in an earlier for-loop segment, referenced only via
`$label` at the point of the verb).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Commit 1596148ff made single_query_deny_message a required keyword-only
parameter of _run_approval_gate() and updated its two callers inside
tools/approval.py, but missed the third caller: the SSH-config write
guard in tools/file_tools.py (_check_approval_required_write,
pattern_key="ssh_config_write").
Any gated write to an SSH client config therefore raised
TypeError: _run_approval_gate() missing 1 required keyword-only
argument: single_query_deny_message
instead of routing through the human-approval flow.
- Pass the kwarg with a single-query-specific deny message that points
operators at approvals.single_query_mode: approve.
- Add a regression test asserting the gate call passes every required
kwarg (fails on unpatched main).
Fixes#93201
The separator-class anchor broke two fail-closed tests (binary bytes
decode to U+FFFD adjacent to the CLI name; remote head-c reads). A
negative lookbehind excluding path/word chars keeps the #77173 fix
while preserving every fail-closed content-scan path.
A file path with embedded spaces (/docs/... with lifecycle words in the
filename) matched Branch A via the path tail and hard-blocked innocent
commands. Anchor the CLI name at command position (start, separator, or
substitution opener). Salvaged from PR #77536 by @eaglezzz0522-cloud,
reapplied onto the current pattern with subshell coverage and tests.
The image-routing vision path calls detect_local_server_type without
the provider's API key. Against a remote API-keyed endpoint (sglang /
vLLM with --api-key) every leg of the 5-request probe waterfall came
back 401 — and because a failed verdict was never written to the
in-memory cache (only positive verdicts were), the waterfall re-ran on
EVERY image-bearing turn (#89863: 51 detail-less busy-acks observed in
one Slack channel while the probe sprayed the user's own server).
Two changes:
- image_routing._should_probe_ollama_vision now takes the API key and
forwards it; a new _resolve_inference_api_key mirrors
_resolve_inference_base_url's resolution order (runtime value,
model.api_key, providers blocks) so the key always matches the URL
being probed.
- detect_local_server_type caches a None verdict in memory with a short
failure TTL (5 min, vs 1h for positives) so the next turn is served
from the negative entry instead of re-running the waterfall — while
a transient failure (server starting, key being fixed) recovers in
minutes. Negative verdicts are deliberately not written to the
cross-process disk cache.
Fixes#89863. With a custom: provider pointing at a remote, API-keyed
endpoint (sglang/vLLM/OpenAI-compat), every image turn triggered a 5-request
probe waterfall without Authorization, spraying 401s at the backend.
Two fixes:
1. _should_probe_ollama_vision now takes api_key and forwards it to
detect_local_server_type so keyed local servers don't 401.
2. When provider != 'ollama', remote endpoints (per is_local_endpoint) are
rejected early — server-fingerprint probing is only valid for local
boxes. Non-Ollama remotes expose Ollama-compat endpoints that can
misidentify and trigger unnecessary /api/show probes.
_lookup_supports_vision resolves the runtime api_key via
_runtime_main_value and forwards it to both helpers. New test class
TestShouldProbeOllamaVision covers the contract in both directions.
Route the max-iterations diagnostic through logging when quiet_mode is active so automation wrappers keep stdout machine-readable.
Add a regression test covering quiet max-iteration summary handling.
CI sibling-test blast radius from the cluster branch:
- singleFlightSessionResume crashed on run() doubles that return
non-promises (Cannot read 'finally'); wrap via Promise.resolve().then(run).
- Three use-session-tile-delegate tests pinned the pre-#92961 ambient
dispatch for default-profile sessions; the routing-authority change
intentionally routes every known owner through the profile router, so
the tests now assert requestGatewayForProfile('default', ...) instead.
Brooklyn Nicholson