The referenced-script walk in cron/lifecycle_guard.py capped each file
(1 MiB) and the recursion depth (8) but not the walk: a command referencing
hundreds of scripts, or one enormous shlex token, held the GIL for minutes
on every gateway terminal call (#78398).
Add a per-walk _LifecycleScanBudget (bytes, lines, longest line, unique
paths, remote reads) charged BEFORE any text reaches shlex, and cap each
referenced read at the remaining byte budget so an oversized file is never
read whole. Exhaustion fails closed (the existing contract for one oversized
file) and is logged at WARNING so operators can tell it from a genuine
lifecycle block. Limits are sized so real wrapper graphs never hit them:
a 200-script benign graph is allowed and a restart hidden behind it is
still caught.
tools/terminal_tool.py gates its optional launchctl pre-scan (which also
tokenizes) on the same budget; the full guard still runs afterwards.
Redesigned from #83821 by @Riccardo-Vecchi, which introduced the budget
idea but blocked benign wide graphs (64-path cap) and bundled a suffix
classification change that is left out here.
Refs #78398
Review follow-up on #84203. Both points reproduce; neither was a regression
from the first pass, but both are live bypasses of the same guard.
**Privilege and namespace wrappers were missing.** The allowlist covered the
coreutils-shaped wrappers but not the privilege ones, so each of these ran a
lifecycle script straight past the walk:
pkexec bash ~/restart.sh
runuser -u root -- bash ~/restart.sh
setpriv --reuid=0 -- bash ~/restart.sh
systemd-run --scope bash ~/restart.sh
nsenter --target 1 --mount bash ~/restart.sh
unshare -r bash ~/restart.sh
Added `pkexec`, `su`, `runuser`, `setpriv`, `systemd-run`, `nsenter` and
`unshare`, each with the value-taking options that would otherwise be
mistaken for the command (`nsenter -t 1`, `systemd-run -p X=1`,
`runuser -u root`, …).
**An option can carry a command STRING, not an argv tail.** `env -S` and
`su`/`runuser` `-c` take shell source. The peel treated the operand as an
opaque value and skipped it, so `env -S 'bash ~/restart.sh'` was never
scanned — the string went unread rather than being recursed into.
`_STRING_COMMAND_OPTIONS` now names those options and their values are
re-scanned as shell source, the same treatment `sh -c` payloads already get.
They are read at the ORIGINAL command token, before the transparent-prefix
peel, because peeling past `su`/`env` would discard the very option carrying
the command. `--opt value` and `--opt=value` are both handled.
Scope, stated plainly: this is an enumerated allowlist, not a general
solution to "wrapper that execs its tail". A wrapper outside the set, or a
value-taking option outside these tables, still resolves to no reference —
that fails open, exactly as it did before this PR, and it is a miss rather
than a false block. The reviewer offered "extend the set with tests, or
document that the list is heuristic"; this does the first and states the
second.
Tests: 23 new cases (220 in the file) — every added wrapper against a script
reference including the value-operand option forms, both command-string
option spellings for env/su/runuser, and the same wrappers around ordinary
work (`pkexec systemctl status nginx`, `su -c 'ls -la'`, `env -S 'echo hi'`,
`nsenter -t 1 -m ps aux`) which must stay allowed. 15 fail on the tree
before this commit.
False positives re-checked at scale: the 9,258 command lines from this
repo's own scripts and docs give an identical verdict set before and after —
0 new false positives, 0 lost detections, 0 exceptions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cSddnhxiUmdGbgyKnpg8p
`_mask_data_sink_arguments` exempts lifecycle text living in the arguments of
executables that cannot run them (`grep`, `rg`, `journalctl`, `sqlite3`, …),
so hunting for a restart string in logs is diagnostics rather than a command.
The exemption is dropped when an argument looks like an escape back into
execution — including anything starting with a dot, because sqlite3 spells
its escapes as dot-commands (`.shell`, `.system`).
But `.`, `./x` and `../x` are ordinary path operands, and
grep -r 'systemctl restart hermes-gateway' .
is the most ordinary recursive search there is. The leading-dot test treated
its `.` operand as a sqlite3 escape, disabled masking for the whole segment,
and blocked the command outright — the exact false-positive class the
exemption exists to prevent, on the shape most likely to hit it. Searching a
relative subdirectory (`./logs`, `../archive`) fails the same way, as does a
relative sqlite3 database path (`sqlite3 ./stats.db "SELECT ..."`).
Require a dot followed by a NAME character (`^\.[A-Za-z]`) so a dot-command
still defeats the exemption while a relative path stays a path. A dotfile
operand (`.env`) still reads as a dot-command — conservative, and unchanged
from today's behavior.
This narrows a security guard in the permissive direction, so the escape
hatches are pinned explicitly: with a relative-path operand present,
`.shell`/`.system`, psql's `\!`, a pipe into `sh`/`bash`/`sudo sh`/`xargs`,
command substitution, and a `;`/`&&` continuation all still block. Only the
segment's own data arguments are masked, and only when nothing in it can
reach execution.
Tests: 18 new cases in tests/hermes_cli/test_gateway_restart_loop.py — the
relative-path shapes that must now be allowed, plus the ten escape-hatch
shapes that must still block. The allow cases fail on the unfixed tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cSddnhxiUmdGbgyKnpg8p
`sudo`, `env`, `nohup`, `timeout` and friends exec their argument tail, so
the command that actually runs sits further right. Three guards read only the
first token of a segment, saw the wrapper, and never inspected what it runs:
bash ~/restart.sh → blocked
sudo bash ~/restart.sh → allowed
launchctl submit -l com.x -- helper → blocked
sudo launchctl submit -l com.x -- helper → allowed
Same foot-gun, one word of prefix. That reaches both enforcement points —
`cron.jobs.create_job` and `tools/terminal_tool.py` under `_HERMES_GATEWAY=1`
— and defeats the label-independent submit block that #62891 added precisely
because a persistent helper is the indirect route to a restart loop.
`_peel_transparent_prefixes()` walks past a bounded chain of these wrappers,
skipping their own options, their value-taking options (`sudo -u deploy`,
`stdbuf -o0`), `VAR=value` assignments, a `--` end-of-options separator, and
`timeout`'s duration operand, then returns the index of the real command. It
is applied to the referenced-script walk, the `sh -c` payload walk, and the
`launchctl submit`/`bootstrap` block.
In the referenced-script walk the peel is ADDITIVE — the segment is read at
the original token and again at the peeled one — because peeling must never
remove a reference the un-peeled read would have found. A local script named
`./timeout` is a script, not the coreutils wrapper, and consuming it as a
prefix would have silently stopped scanning it. (The other two call sites
need no such care: no wrapper name is also a shell name or `launchctl`, so
peeling there can only add.) That split is why the per-index logic now lives
in `_references_at()`.
This is not a new reading of shell syntax for this module — `_PIPE_TO_INTERPRETER`
already treats `sudo ` as transparent for the pipe case (`... | sudo sh`).
This generalises the same reading to the command position.
Deliberately NOT applied to the data-sink masking in
`_mask_data_sink_arguments`: peeling there would widen an exemption, and the
conservative reading is the safe one.
No false positives: peeling only changes which token is treated as the
command, so a wrapper around ordinary work resolves to a non-shell executable
and yields nothing, exactly as before (`sudo apt-get update`,
`timeout 60 curl ...`, `nice -n 10 make -j4`, a bare `env`).
Tests: 40 new cases in tests/hermes_cli/test_gateway_restart_loop.py — every
wrapper form against a script reference, a dot-source, a nested `sh -c`
payload and `launchctl submit`, plus the benign wrapped commands, a wrapped
clean script, and the `./timeout`-style lookalike names that pin the additive
reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cSddnhxiUmdGbgyKnpg8p
The #76762 binary check treats any NUL byte in the first chunk as "compiled
binary, nothing to scan":
if b"\x00" in data:
return None, False
"Contains a NUL" and "is a compiled binary" are different questions, and the
gap between them is a guard bypass. `bash` executes a *text* script straight
past an embedded NUL, so one pad byte disables the entire scan while the
script still runs:
#!/bin/bash
# pad<NUL>
hermes gateway restart
scan("bash padded.sh") -> False (not blocked)
bash padded.sh -> executes the lifecycle command
This shape was blocked before #76762, so the crash fix traded a loud failure
for a silent one.
Keying the check on a leading `#!` is not sufficient: a shebang-less file with
a NUL on any line but the first also executes normally. (A NUL on line 1 of a
shebang-less file is the one shape bash rejects, exit 126 — but that same file
is still executable via `. file`.)
Fix: identify binaries by MAGIC NUMBER — ELF, Mach-O (incl. byte-swapped and
universal/fat), PE/COFF, static archive, gzip, zip — with a shebang always
winning. A NUL-bearing *text* file is scanned with its NULs stripped;
stripping can only splice tokens together, never apart, so it fails closed.
File extensions are deliberately not consulted, so a suffixless shell script
is still scanned.
The size check now runs BEFORE the strip: stripping shrinks the buffer, so
checking afterwards would let an oversized file slip under the threshold and
skip the fail-closed branch. (Caught by
test_oversized_nul_bearing_text_still_fails_closed, which failed on the first
cut of this patch.)
Return values are unchanged, so this does not conflict with the in-flight
crash-class fixes to the same function.
Tests (tests/hermes_cli/test_gateway_restart_loop.py), 3 of which fail on main:
- test_nul_padded_script_is_still_scanned
- test_nul_padded_script_without_shebang_is_scanned
- test_oversized_nul_bearing_text_still_fails_closed
- test_elf_binary_is_not_scanned_as_script (#76762 stays fixed)
- test_macho_binary_is_not_scanned_as_script (incl. fat binary)
- test_clean_script_without_lifecycle_command_not_blocked
`_iter_referenced_shell_scripts` recognises the `source` builtin so a script
pulled in with `source ./restart.sh` gets scanned for lifecycle commands. The
POSIX dot operator is the same builtin, but it was not caught:
if executable_name in {".", "source"}:
`executable_name` is `Path(executable).name`, and `Path(".").name` is the
**empty string** -- pathlib normalises "." to the current directory, whose name
is "". So the set membership never matched for `.`, the sourced script was
never added to the reference walk, and its contents were never scanned.
Verified against current main:
. /tmp/restart.sh -> not blocked (script never scanned)
source /tmp/restart.sh -> blocked
bash /tmp/restart.sh -> blocked
where /tmp/restart.sh contains a `hermes gateway restart` line. Sourcing runs
the script in the current shell, so the dot spelling is not merely equivalent
to `source` -- it is the more common form in practice.
Fix compares the raw token as well as the basename:
if executable in {".", "source"} or executable_name == "source":
Keeping the `executable_name == "source"` arm preserves the existing behaviour
for a path-qualified spelling, while the raw-token test catches `.` without
relying on pathlib normalisation.
Tests (tests/hermes_cli/test_gateway_restart_loop.py):
- test_dot_operator_sourced_script_is_scanned -- the regression; fails on main
- test_source_builtin_sourced_script_is_scanned -- `source` stays blocked
- test_dot_operator_clean_script_not_blocked -- widening the check must not
false-block an innocent `. ./activate.sh`
Found while auditing the guard after #76762. Scoped deliberately to this one
defect; the NUL-padded-script bypass I found in the same audit is a separate
PR.
Runbook prose inside a quoted-delimiter heredoc feeding a data sink
(cat > file <<'EOF') is documentation, not a command this shell will
execute. Mask provably-inert heredoc bodies (tools/shell_heredoc's
conservative stripper, already used by terminal_tool) before scanning.
Fails open on any ambiguity: executable and unquoted-delimiter heredocs
stay scanned. Salvaged from PR #88336 by @zgqq (the heredoc half; its
Branch D boundary and dir-token halves already landed/were fixed).
execute_code lacked the lifecycle guard entirely, and Python argv-list
forms (subprocess.run([...])) separated command words with brackets and
commas the shell-shaped pattern could not see. Mirror the terminal_tool
guard in execute_code (ownership-gated per #92560) and strip argv-list
punctuation in the token-join re-scan. Salvaged from PR #68289 by
@arcimun, adapted to the ownership gate and current guard structure.
The gateway-lifecycle guards in cron/lifecycle_guard.py (Branch B, the
unconditional hard-block used by cron creation and the terminal tool when
_HERMES_GATEWAY=1) and tools/approval.py's launchctl rule both matched
`launchctl <verb> ... hermes[.-]?gateway` as a single sequential regex,
requiring the hermes-gateway label to appear literally AFTER the verb.
A shell command that builds the label earlier in the string — e.g. a
for-loop reading labels from a list defined before the actual launchctl
call — defeats that ordering entirely:
for item in 'ai.hermes.gateway-apollo:...' 'ai.hermes.gateway:...'; do
label=${item%%:*}; plist=${item#*:}
launchctl bootout "gui/$uid/$label"
launchctl bootstrap "gui/$uid" "$plist"
done
The literal text "hermes.gateway" only ever appears in the for-list,
never after "bootout" — so `[^\n]*\bhermes[.\-]?gateway` never matches at
the verb's position, even though the command unambiguously targets the
gateway's own launchd label.
cron/lifecycle_guard.py's verb list also didn't include `bootout` at all
(present in tools/approval.py's list and covered by its own test suite —
`launchctl bootout ai.hermes.gateway` is explicitly asserted as dangerous
there — so the omission in the sibling file looks like list drift between
the two guards rather than an intentional exclusion).
`bootout` is the verb that actually deregisters a launchd job (unlike
kickstart/stop, which just bounce a still-registered one), so a command
using it evades both guards, then removes the service from launchd with
no supervisor left to bring it back — worse than a simple restart-loop.
We hit this for real: a gateway self-restart (triggered from a chat
request to change the default model) used a raw terminal `launchctl
bootout`/`bootstrap` loop across 4 launchd labels instead of the normal
`hermes gateway restart` path. It slipped past both guards, self-bootout
killed the process mid-drain before its own follow-up bootstrap could
run, and all 4 gateway profiles ended up fully deregistered from launchd
with zero user approval (approvals.mode: manual was configured) until
someone manually re-bootstrapped them.
Fix: both guards now check "a launchctl lifecycle verb appears somewhere
AND a hermes-gateway label appears somewhere", independent of order, and
cron/lifecycle_guard.py's verb list gains bootout/kill/disable/remove to
match tools/approval.py's existing set. Internal recovery code
(hermes_cli/gateway.py's own `subprocess.run(["launchctl", "bootout",
...])` calls) is unaffected — these guards only scan shell-command
strings composed by the agent's terminal/cron tools, not the CLI's
trusted internal subprocess argument lists.
Adds regression tests in both test files reproducing the exact incident
command (label built in an earlier for-loop segment, referenced only via
`$label` at the point of the verb).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The separator-class anchor broke two fail-closed tests (binary bytes
decode to U+FFFD adjacent to the CLI name; remote head-c reads). A
negative lookbehind excluding path/word chars keeps the #77173 fix
while preserving every fail-closed content-scan path.
A file path with embedded spaces (/docs/... with lifecycle words in the
filename) matched Branch A via the path tail and hard-blocked innocent
commands. Anchor the CLI name at command position (start, separator, or
substitution opener). Salvaged from PR #77536 by @eaglezzz0522-cloud,
reapplied onto the current pattern with subshell coverage and tests.
The hard block matched raw command text, but a shell resolves quote
splicing (`kick"start"`) and backslash escaping (`kick\start`) into the
literal verb before execution. So `launchctl kick"start" -k
gui/501/ai.hermes.gateway` ran exactly as the blocked `kickstart` form
while both the non-bypassable block and the approval detector missed it —
leaving an approval-bypassing gateway self-lifecycle operation reachable.
contains_gateway_lifecycle_command now runs a second pass over
shlex-tokenized command segments, where quotes and escapes are already
resolved. It stays anchored on a hermes-gateway identifier, so prose and
non-gateway hermes services are unaffected. Because this function is the
single choke point _contains_unsafe_gateway_action calls at every
recursion level, referenced-script and `sh -c` payload scanning inherit
the fix.
tools/approval.py had the same gap for quote splices: backslash escapes
are stripped by _normalize_command_for_detection, but quote splicing in an
ARGUMENT position is not touched by _deobfuscate_shell_word_for_detection
(scoped to command-position words, deliberately — widening it would let
quoted prose match the destructive patterns). It now delegates to the
fixed guard as a last check, so an ordinary pattern match still wins and
keeps its more specific reason string.
Tests: quoted, single-quoted and backslash-spliced verbs across the
launchctl/systemctl/hermes branches, the spliced gateway identifier
itself, a splice nested in an `sh -c` payload (resolves one level deeper,
asserted at the recursive entry point terminal_tool actually calls), plus
negative cases proving prose and non-gateway labels stay unblocked.
Verified on Windows: no regressions — the 10 remaining failures across
tests/tools/test_approval.py, tests/hermes_cli/test_gateway_restart_loop.py
and tests/cron are identical on the unmodified baseline (POSIX file modes,
symlink privileges, and /bin/bash script paths).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prose false positives (Branch A trailing boundary, Branch D leading
boundary), quoted-multiline data payloads, and the fail-closed
unbalanced-quote fallback.
Test-helper + regression test from PR #92633 by @DavidMetcalfe:
mock _is_supervised_gateway_process instead of setting the raw env
var, and add a test proving a CLI agent session with inherited
_HERMES_GATEWAY=1 but no PID ownership is no longer blocked.
When check_gateway_lifecycle refuses a cron script that lives on a
FileProvider path, the generic error implied the job contained a dangerous
gateway lifecycle command. Surface the real reason instead — the script
lives on a cloud-synced path whose evicted placeholder could hang the
preflight scan — while staying fail-closed. Regression test asserts the
cron-script scan path blocks without opening the file and that the message
names the cloud-synced path rather than a lifecycle command.
Widen #88052 per review:
- The walk-level short-circuit only protected _contains_unsafe_gateway_action;
the sibling caller _read_script_for_scanning still opened cloud-resident cron
scripts and could hang preflight. Move the check into _read_referenced_script,
the shared choke point, so every caller fails closed without opening.
- Generalize _is_apple_file_provider_path -> _is_cloud_placeholder_path: detect
~/Library/CloudStorage (Dropbox/OneDrive/Google Drive third-party FileProvider
domains) alongside iCloud's Library/Mobile Documents.
- Regression tests: CloudStorage lexical path blocked without open; the choke
point itself refuses cloud paths with os.open forbidden.
Docker Desktop writes fpath=(~/.docker/completions ...) into .zshrc.
The referenced-script walk then opened that directory, saw a non-regular
file, and fail-closed — blocking source ~/.zshrc on every terminal
command. Directories are not scripts; devices stay fail-closed.
The auto-resume restart-loop breaker (#30719, defense-3) pruned its boot
log against an absolute `window_seconds` (default 60s). That prune is
period-sensitive: a crash cycle slower than the window drops its own
history on every boot, so the counter never leaves 1 and the breaker can
never trip, no matter how long the loop runs.
The cycle reported in #81642 is ~150s — a wedged event loop, the liveness
watchdog hard-exiting at ~90s, a supervisor respawn, and auto-resume
replaying the same session that wedges it again. Structurally invisible to
a 60s window: `gateway/restart_loop.json` kept a single timestamp across 15
kills in one morning. Because every cycle leaves a gateway that cannot
process SIGTERM, `hermes update` has no drainable gateway to stop, which is
the reported hang.
Chain boots on the inter-boot GAP instead of an absolute window: two boots
belong to the same loop when they are no more than `max_gap_seconds` apart
(default 300s, floored by `window_seconds` so widening the window never
makes the breaker less sensitive). The verdict becomes period-agnostic —
the original ~10s respawn loop still trips in 3 boots, and so does a 150s
one — while a boot after real quiet resets the chain, so occasional
operator restarts still never accumulate. The persisted chain is capped at
50 entries.
- gateway/restart_loop_guard.py: gap-chained pruning (`_chain_ending_at`),
`DEFAULT_MAX_GAP_SECONDS`, `max_gap_seconds` kwarg on the three entry
points, clock-step tolerance, bounded state file
- gateway/run.py: `_restart_loop_guard_config` reads and returns
`max_gap_seconds`; the auto-resume call site passes it through
- hermes_cli/config_defaults.py: `gateway.restart_loop_guard.max_gap_seconds`
Tests: 7 new cases in TestRestartLoopGuard covering the slow cycle, chain
persistence, quiet-period reset, the #30719 fast loop, the config knob, the
window floor, and the disabled breaker. Verified RED before the fix (the
slow-cycle case asserted `[1300.0] == [1000.0, 1150.0, 1300.0]`, exactly
the single-timestamp state file from the report) and GREEN after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two live failures on the same guard (cron/lifecycle_guard.py), both of
which blocked legitimate diagnostics from inside the gateway:
1. Crash class: the referenced-script walk read compiled binaries as if
they were shell scripts. Reading/inspecting a referenced file is now
best-effort by construction: executable magic numbers (ELF, PE,
Mach-O fat/thin) short-circuit before any full read via a 4KB sniff,
NUL-bearing heads are skipped as non-scripts, and unreadable paths of
every kind (NUL bytes in the token, ENAMETOOLONG, missing files)
degrade to "nothing to scan" instead of raising. A second fail-safe
layer wraps the pure-string fallback so the boundary function stays
total even if the tokenizer itself fails.
2. False-positive class: the lifecycle regex matched its command shapes
inside DATA arguments — SQL string literals passed to sqlite3/psql
and grep/rg/journalctl patterns hunting for the lifecycle string in
logs. Added a fail-closed second-pass exemption: on a raw regex hit,
re-scan with data-sink executables' arguments masked; only a match
that survives (i.e. sits in command position) blocks. Masking is
skipped for pipes into shells/xargs, command/process substitution,
sqlite3 dot-commands and psql backslash escapes, so it can only ever
allow, never miss.
Behavioral tests: exact live false-positive shapes as negatives, the
smuggling shapes as positives, the kill-primitive positive catalog
unchanged, and an adversarial never-raises suite (NUL bytes, non-UTF-8,
/dev/*, directories, missing files, magic-prefix binaries).
3-reviewer simplify pass (reuse/quality/efficiency) findings:
- cron/scheduler.py _run_job_script: the ORIGINAL that
lifecycle_guard._resolve_script_path documents mirroring had the exact
same unguarded expanduser() — a NUL-bearing script value survives
creation (the guard treats it as nothing-to-scan) and crashed the
scheduler at fire time with ValueError instead of a clean job failure.
Same ingestion contract applied; regression test added.
- lifecycle_guard._resolve_script_path: get_hermes_home() -> Path.home()
raises RuntimeError when neither HERMES_HOME nor HOME resolves
(arbitrary-UID containers); the cron entry point called it bare.
Caught -> None; totality test added.
- terminal_tool: stale 'cat ...' docstring updated to the bounded
head -c form.
- lifecycle_guard: dead 'script_text and' condition dropped (guarded by
'if not script_text: continue' directly above).
Efficiency reviewer: no material findings (measured — encode/expand
costs negligible vs walk I/O, no timing regression vs base).
The guard feeds untrusted byte streams (tokenized binaries, remote cat
output) into OS-path and shell-text operations; every incident so far
(#76762, #77703, #77780, #78256, #77729) was hot-fixed with an except at
whichever frame crashed that week. tilllt's regression suite on #79454
showed 4 members of the class still open on merged main. Close the class
at three boundaries instead:
- _expand_candidate_path(): single ingestion chokepoint for path
candidates — reject NUL/empty tokens before any Path OS call and
tolerate ValueError/RuntimeError/OSError from expanduser (T1/T2, plus
the HOME-unset launchd crash). Both _resolve_terminal_script_path and
_resolve_script_path now go through it.
- _sanitize_remote_script_text(): apply the local-read contract (NUL =
binary = nothing to scan; >1MiB = fail closed) to whatever any
read_remote_script callback returns, at the recursion boundary — the
guard stops trusting its callbacks (T3/T4).
- contains_gateway_lifecycle_command_or_referenced_script() is now total
by construction: direct regex scans (pure string ops) run first; the
best-effort filesystem walk is wrapped so an unexpected failure logs a
warning and falls back to the direct-scan verdict instead of killing
every terminal command until gateway restart.
terminal_tool's remote fallback also bounds the read at the source
(head -c 1MiB+1 instead of cat), so a 166MB ELF never crosses the wire —
the superlinear-shlex 30-minute stall from #79838's field report drops
to a 0.02s fail-closed verdict.
Regression tests: tilllt's T1-T4 adopted verbatim, plus an adversarial
never-raises sweep (NUL paths, unset HOME, over-long paths) and a
walk-crash fallback test.
Residual #76762 class: _read_referenced_script caught OSError from
os.open but not ValueError, so a path token carrying an embedded NUL
(tokenized binary-adjacent command text) crashed the terminal tool's
lifecycle guard with 'ValueError: embedded null byte' instead of being
skipped as nothing-to-scan. Reproduced live against main. Same
treatment the resolve()-time site already has; two sabotage-verified
regressions added.
The gateway terminal guard crashed with 'ValueError: embedded null byte'
(command never ran, exit_code -1) when a command invoked an ELF binary by
full path. _read_referenced_script correctly rejects the binary locally
(NUL in first chunk), but the read_remote_script fallback
(_read_script_in_env) then re-read the SAME file's bytes without a NUL
guard, decoded them, and fed machine code back into the scanner, which
re-tokenized it into a bogus NUL-bearing path and crashed at os.open.
- _read_script_in_env: skip content containing a NUL byte on both the
local-read and remote-cat branches (mirrors _read_referenced_script:
a binary is nothing to scan), so binary never re-enters the guard.
- _read_referenced_script: tolerate ValueError from os.open on a
NUL-in-path, alongside the existing OSError guard, so the guard can
never crash the terminal tool regardless of input.
Extends the #76762 NUL-safety fix (local path only) to the gateway's
remote-read fallback path.
The gateway lifecycle guard (cron/lifecycle_guard.py) applied shell-style
tokenization and script-reference resolution to non-shell content, with two
regressions:
#77131 - every .py cron script using pathlib division was hard-blocked:
Path.home() / ".hermes" / ".env" tokenizes the bare "/" operator as an
executable path, which resolves to the filesystem root; the regular-file
check then fails closed as unsafe. Since Python runs under the
interpreter, never through a POSIX shell, the shell-script reference walk
is a false-positive generator on Python sources. check_gateway_lifecycle
now skips the walk for *.py scripts (the direct command regex still scans
the full text), and _iter_referenced_shell_scripts skips pure-separator
tokens.
#76762 - terminal commands invoking a binary by absolute path (e.g.
/usr/bin/python3) crashed the guard with ValueError: embedded null byte:
the walk read the binary's bytes, decoded them as text, and re-tokenized
machine code; the recursion then hit Path.resolve() on a NUL-bearing
path while only OSError was caught. _read_referenced_script now skips
NUL-containing files (binaries are not referenced shell scripts) and
resolve() tolerates ValueError.
Shell scripts (.sh/.bash/.zsh) keep the full deep scan; literal lifecycle
commands in .py scripts are still blocked by the direct regex. New tests
cover all four behaviors.
Extends the shared _GATEWAY_LIFECYCLE_PATTERN (used by BOTH the cron
creation-time guard in cron/lifecycle_guard.py and the terminal
execution-time hard-block in tools/terminal_tool.py) so Branch B covers
launchctl submit and bootstrap alongside kickstart/unload/load/stop/
restart, and normalizes POSIX shell line continuations before matching
so the exact multi-line reported shape in #62891 cannot slip past.
Also extends the execution-aware, label-independent detector
(contains_launchctl_submit_command, cherry-picked from #63272) to cover
launchctl bootstrap, since a neutral label like ai.hermes.svc-reload-tmp
defeats any label-anchored regex — the second production reproduction.
Regression tests cover both sites, including
`launchctl submit -l com.foo -- /path/gateway` and the bootstrap
variant, plus outside-gateway pass-through.
Branch B regex extension and continuation normalization drawn from
PR #62896; bootstrap coverage and test shapes drawn from PR #51003.
Co-authored-by: JackJin <1037461232@qq.com>
Co-authored-by: joelbrilliant <joelbrilliant1@gmail.com>
- Resolve guard cwd against get_session_cwd(session_key); fall back to env.cwd
when no session record exists yet, matching current main's per-session cwd
architecture.
- Make referenced-script reads backend-aware: local read first; if missing,
fall back to env.execute('cat ...') for SSH/Modal/Daytona backends.
- Reuse the recursive scanner in check_gateway_lifecycle so nested cron
wrapper scripts are caught, and resolve relative refs inside a script
against that script's directory.
- Add regression tests for remote-backend reads, two-session cwd, and nested
cron wrappers.
Verification: 80 passed tests/hermes_cli/test_gateway_restart_loop.py;
694 passed tests/cron; ruff + git diff --check clean.
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
Completes the #30719 restart-loop defenses. Defenses 1-2 (the
_HERMES_GATEWAY guard on `hermes gateway stop|restart` + terminal_tool,
and the cron-creation lifecycle filter) already landed on main, but two
gaps remained:
- The agent's `cronjob` model tool calls cron.jobs.create_job directly,
bypassing the hermes_cli.cron.cron_create CLI filter, so lifecycle
commands scheduled via the model tool were only blocked at execution
time (terminal_tool), not at creation. Moved the filter to a shared
cron/lifecycle_guard.py enforced at create_job — the single chokepoint
every job-creation path hits (CLI + model tool). Re-exported
_contains_gateway_lifecycle_command from hermes_cli.cron so
terminal_tool's import keeps working.
- No breaker for the auto-resume loop itself. Defenses 1-2 cover the
cron/CLI/terminal paths, but any other SIGTERM source (e.g. a raw
terminal("launchctl kickstart ai.hermes.gateway")) still triggers the
boot->auto-resume->re-run cycle. Added gateway/restart_loop_guard.py:
counts restart-interrupted boots in a rolling window (config
gateway.restart_loop_guard, default 3 boots / 60s) and skips
auto-resume for that boot once tripped. The gateway still comes up and
serves real inbound messages; it just stops replaying the session that
keeps killing it, putting a human back in the loop.
Also tightened the lifecycle regex over main's version: dropped
`hermes gateway start` (benign), required the gateway identifier on the
launchctl/systemctl branches (so `launchctl unload
ai.hermes.update-checker.plist` and `systemctl restart
hermes-meta.service` no longer false-positive), added the inverse
pkill token order, and fixed the binary-script bypass (decode with
errors='replace' instead of swallowing UnicodeDecodeError). The
create_job guard resolves relative script paths under HERMES_HOME/scripts
the same way the scheduler does, so a bare script name is scanned as the
file that actually runs.
Design and much of defense-2 originate from PR #33395 (@kshitijk4poor),
which itself salvaged #30728 (@SimoKiihamaki). Rebuilt against current
main since defenses 1-2 had already landed under different names.
Closes#30719.
Co-authored-by: SimoKiihamaki <simo.kiihamaki@gmail.com>
Co-authored-by: kshitijk4poor <82637225+kshitijk4poor@users.noreply.github.com>
* fix(terminal): require approval for host-bound Docker commands
The Docker terminal backend blanket-skips dangerous-command approval on
the assumption that the container is isolated from the host. That holds
only when nothing is bind-mounted in. Once a host path is exposed (via
TERMINAL_DOCKER_MOUNT_CWD_TO_WORKSPACE or a host-path entry in
TERMINAL_DOCKER_VOLUMES), a command like `rm -rf /workspace` reaches
real host files but is still auto-approved.
Detect host bind mounts and route those sessions through the normal
approval flow. Isolated Docker keeps the fast path. The same gating is
applied to the execute_code guard, which had the identical blanket skip.
Co-authored-by: Hermes Agent <agent@nousresearch.com>
* chore: add AUTHOR_MAP entry for PR #6436 salvage (Kolektori)
* test: accept has_host_access kwarg in _check_all_guards mocks
The host-bound Docker approval fix adds a has_host_access kwarg to the
_check_all_guards wrapper. Six pre-existing tests monkeypatch it with a
fixed (command, env_type) / (cmd, env) lambda signature, which now
raises TypeError when terminal_tool passes the new kwarg. Widen those
mock signatures to accept **kwargs.
---------
Co-authored-by: Kolektori <256073454+Kolektori@users.noreply.github.com>
Co-authored-by: Hermes Agent <agent@nousresearch.com>
systemctl --user restart hermes-gateway run via the terminal tool is a
child of the gateway itself. When systemd delivers SIGTERM the gateway
kills this subprocess before it can complete, so the service may never
restart — reproducing issue #37453.
The hermes gateway restart/stop guard (hermes_cli/gateway.py) and the
cron-path guard (hermes_cli/cron.py) already block equivalent commands
in their respective paths but the terminal tool had no such defense.
Add a hard-block before command execution in terminal_tool: when
_HERMES_GATEWAY=1 and the command matches _contains_gateway_lifecycle_command,
return an error immediately. force=True cannot bypass it — unlike the
normal dangerous-command approval flow, here even a user-approved restart
would fail because the SIGTERM propagates to child processes.
Also extend _GATEWAY_LIFECYCLE_PATTERNS to match systemctl with flags
(e.g. systemctl --user restart) — the previous regex required the
action word immediately after systemctl with no flags in between.
Adds 9 regression tests: 6 blocked variants (parametrized), force bypass
attempt, safe systemctl passthrough, and guard-inactive-outside-gateway.
Follow-up to the salvaged #30728:
- Gateway already exports _HERMES_GATEWAY=1 at startup (gateway/run.py) and
cli.py already keys off it. Drop the redundant new HERMES_IN_GATEWAY var;
guard stop/restart on _HERMES_GATEWAY instead. One marker for one fact.
- Drop the greedy \bgateway.*restart alternation from the cron lifecycle
filter — it false-positived on legit prompts that merely mention an
unrelated gateway + a restart (API/payment gateway monitoring). The
specific 'hermes gateway (restart|stop|start)' pattern already covers the
real command.
- Rework the two negative guard tests to sentinel the first downstream call
so they don't drive real signal delivery (tripped the live-system guard).
- Add false-positive regression cases to test_safe_commands.
Three defenses against SIGTERM-respawn loops when agent schedules its
own gateway restart under launchd/systemd KeepAlive:
1. HERMES_IN_GATEWAY env var: gateway sets it at startup; stop/restart
subcommands refuse to run when set (exit 1 with clear message).
2. Cron create payload filter: regex pre-flight rejects prompts/scripts
containing hermes gateway restart/stop, launchctl kickstart/unload,
systemctl restart/stop, and pkill patterns.
3. 30 new tests: pattern matching (14), cron block (5), gateway guard (4),
safe command negatives (7).
Teknium