Six test files still selected an OS branch with a faked host. Each one now
carries the marker for the host that owns the branch, or derives the
expectation from the real host:
- test_clipboard: macos_only on the has_clipboard_image dispatch. The fake
picked the branch, but _macos_has_image needs osascript.
- test_claw: windows_only on the tasklist/powershell scan, with return_value
in place of a side_effect list that pinned the call count.
- test_linux_desktop_entry: the parametrize over "darwin"/"win32" becomes one
marked test per host. A fake left POSIX paths and a POSIX XDG layout.
- test_graphical_browser_detection: linux_only on the display-server arm. The
$BROWSER check runs before the platform branch, so its test stays unmarked.
- test_auth_nous_provider: the fixture pinned linux so the macOS certifi
fallback could not change the result. The assertion now reads the host, so
the macOS lane covers the fallback too.
- test_tts_macos_output and test_voice_mode: the afplay policy exists because
CoreAudio init raises a TCC prompt, which no Linux runner reproduces.
tests/conftest.py refuses collection when one test carries two OS markers.
Each marker skips on all but one host, so two of them make a test that runs
nowhere while every lane reports green. tests/test_os_marker_gating.py pins
that behavior.
The docstring on TestConfirmDestructiveSlash said the Windows job runs it.
The class has no marker, so -m windows_only deselects it.
many tests patched sys.platform or a module's _IS_WINDOWS flag, then
ran on linux ci. the patch selects the branch under test, but the host
does not have the behavior the branch exists for. the test proves the
patch, not the platform. some gated assertions never ran on any host.
this commit adds three markers: linux_only, macos_only, windows_only.
a conftest hook skips a marked test on the other hosts, with a clear
reason. no test fakes a host now. two documented fakes remain
(android/termux, freebsd) because no ci runner exists for them.
each fake site got one of four treatments:
- gate it: the real host supplies the platform; mocks cover real
dependencies only, never host identity
- patch the module's own probe when the subject is the probe's consumer
- assert against the real host when the fake stood in for any non-x host
- delete the patch when it set the value the host already has
bare skipif(sys.platform != ...) guards became markers too. the lane
model skips these on linux and never imports them on windows, so they
ran on no host. platform parametrize tables are now one marked test
per os.
running on real hosts found real errors: a chrome-sandbox failure in
test_gui_command that main hides, and two windows failures fixed here.
the agents.md testing section now documents the policy.
Forensics on a live developer machine found pytest fixture rows inside the
REAL ~/.hermes/state.db — sessions with chat_id 'chat-1', '123', 'wx-chat',
and gateway_routing rows whose scope was literally under /tmp/pytest-of-*/.
A pytest-spawned process also opened the live DB and flipped its journal
mode (journal_mode=DELETE fallback on SQLite 3.50.4) under the WAL-mode
gateway writer, destroying committed transcripts ("Persisted transcript
lagged live cached history ... possible FTS write corruption", 15+
occurrences). The existing live-system guard covers kill primitives but not
the SessionDB/SessionStore write paths.
Root cause (leak vector): the session-level HERMES_HOME sandbox in
tests/conftest.py only created a tempdir when HERMES_HOME was UNSET. On a
machine where the shell (e.g. gateway-launched, or an exported
HERMES_HOME=~/.hermes) hands pytest the production home, the sandbox was
skipped entirely — every argless SessionDB()/SessionStore() and every
collection-time DEFAULT_DB_PATH froze onto the real state.db.
Fixes (fail the class, one owner):
* hermes_state._ensure_test_isolation(): single choke point wired into
SessionDB.__init__ (every construction, incl. read_only). Under pytest
(PYTEST_CURRENT_TEST / PYTEST_VERSION — inherited by subprocess
children), a db path resolving to <real-root>/state.db or
<real-root>/profiles/<name>/state.db raises RuntimeError('live-system
guard: ...') before any connection, mkdir, or journal-mode pragma.
* tests/conftest.py: session sandbox now also tempdir-redirects a pre-set
HERMES_HOME that points at the production root (the actual escape
vector); kanban deny-list capture updated to match. New autouse
_state_db_write_guard fixture honors the existing
@pytest.mark.live_system_guard_bypass marker as the escape hatch and
feeds custom (non-~/.hermes) production roots into the guard deny-list.
* gateway/session.py: SessionStore.__init__ no longer swallows the guard's
RuntimeError into the JSONL fallback — guard trips are loud.
* tests/hermes_state/test_live_db_isolation_guard.py: behavioral
regression tests — production paths (direct, profile, read-only,
unnormalized, default-resolution) raise; tmp HERMES_HOME works; bypass
marker works; SessionStore re-raises guard errors but still degrades on
ordinary failures; subprocess child without HERMES_HOME is refused while
a hermetic child succeeds.
No new HERMES_* env vars; no hardcoded ~/.hermes (platform root comes from
hermes_constants._get_platform_default_hermes_home()).
Cross-file leaks made tests/tui_gateway + tests/test_tui_gateway_server.py
fail when run in one process (issue #57068):
- conftest: _hermetic_environment now re-pins hermes_state.DEFAULT_DB_PATH
to the fake home so no test ever touches the developer's real state.db
- conftest: new autouse _reset_tui_gateway_server_state snapshots/restores
_methods, cfg cache, db handle and _real_stdout, and tears down leftover
sessions via the production _close_session_by_id(..., end_reason=
"test_cleanup") boundary
- per-file fixtures scope patch.dict(sys.modules) to the import only
- drop reload()-based teardowns that duplicated atexit hooks
Conflict resolution vs main: kept main's mod._live_transports.clear() in
the test_protocol.py server fixture alongside the snapshot/restore logic.
Combined run now 792 passed / 0 failed.
Salvaged from PR #57066 by @lEWFkRAD. Fixes#57068.
Guard the .lazy-refresh-incomplete marker writer (update_cmd), launch-time
recovery (main.py), and _early_recovery repair paths behind a two-condition
check: running under pytest AND the target is this live checkout. Sandboxed
tmp_path tests still exercise the real code paths.
Salvaged from PR #72002 by @fcavalcantirj. Fixes#72000.
Co-authored-by: fcavalcantirj <felipe.cavalcanti.rj@gmail.com>
The guard's sys.modules.get() can observe hermes_cli.kanban_db while a
lazy import is still executing on a fixture boundary — the partially
initialized module has no .connect yet (AttributeError flake in the
full-suite verification run, 1/2460 files). A half-imported module has
no callers to guard; skip this round and let the next fixture patch the
completed module.
hermes_cli/main.py calls setup_logging() at module scope. That resolves
get_hermes_home() and attaches rotating file handlers to the ROOT logger via
a QueueHandler. So merely importing it - which many test modules do, directly
or transitively - points the whole pytest session's logging at
<HERMES_HOME>/logs/agent.log and errors.log.
The _isolate_env fixture already sandboxes HERMES_HOME, but fixtures run
after collection has imported the test modules, and by then the handler holds
an absolute path to the real file. Verified by importing hermes_cli.main in a
clean interpreter and walking the queue listener: both handlers pointed at the
developer's own ~/.hermes/logs/.
Measured on a live install: 126 warnings in a personal agent.log came from
test runs rather than the running gateway - phantom 'FakeTree' Discord
registration failures and 'rejected invalid API key' entries whose paths only
exist in tests/gateway/test_api_server_runs.py. That noise makes genuine
warnings hard to find exactly when someone is debugging.
conftest is imported before any test module, so sandboxing HERMES_HOME there
closes the window. The per-test fixture still applies afterwards.
Also fixes 4 pre-existing failures: tests/gateway/test_channel_directory.py
TestBuildFromSessions was reading the operator's real sessions data for the
same reason.
Full gateway+tools suites: 66 failures on clean origin/main, 62 with this
change, 0 new. The regression guard asserts the value captured AT conftest
import - reading os.environ inside a test passes even with the fix removed,
because the per-test fixture has sandboxed it by then.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Re-port of PR #35464 onto the rewritten hermetic conftest:
- autouse _neutralize_webbrowser fixture records open/open_new/open_new_tab
and webbrowser.get() instead of launching a real browser
- autouse _neutralize_macos_keychain_creds defaults the Anthropic Keychain
reader to None, with an opt-in allow_macos_keychain marker
- regression tests in tests/test_hermetic_side_effect_guards.py
- tests/agent/test_anthropic_keychain.py opts in via pytestmark
Salvaged-from: #35464
Co-authored-by: y0shualee <yuxiangl490@gmail.com>
Autouse conftest fixture patches kanban_db.connect to refuse writes whose
resolved DB path lands under the REAL kanban root (captured at conftest
import time, before fixtures rewire the environment). Deny-list, not
allow-list, so hermetic tests moving HERMES_HOME to sibling tempdirs are
unaffected. Lazily attaches only when hermes_cli.kanban_db is already in
sys.modules.
Salvaged from PR #69385 by @smfworks; rebased by hand onto the pruned
conftest and adapted to guard on the resolved DB path (explicit db_path
or kanban_db_path()) rather than kanban_home() alone.
Co-authored-by: Jasmine Naderi <jasmine@smfworks.com>
tools/computer_use/tool.py keeps the CLI approval flow in module-globals:
_approval_callback plus the per-session unlock stores _always_allow /
_session_auto_approve. Any test that installs a callback (or drives CLI
init far enough that the real one is registered) and does not reset it
poisons every later computer-use test in the same process:
* a leaked callback that raises — dead UI infra or a stale two-argument
signature (the contract is (action, args, summary)) — becomes
verdict='deny' in _request_approval, so dispatch tests fail with an
empty backend call list;
* a leaked callback that blocks (the real CLI one waits on an answer
queue) hangs a single-process run forever.
Both are order-dependent: tests/tools/test_computer_use.py passes 220/220
in isolation but shows dispatch failures in single-process full-suite runs
(and, with a blocking leak, a permanent hang observed via py-spy inside
_request_approval -> callback -> queue.get with no timeout).
Fix: an autouse teardown-only fixture resets callback + unlock stores
after every test; tests that install their own callback keep it for their
own duration. Regression pair included: a 'forgetful' test leaves a stale
two-arg callback behind, the next test asserts dispatch still routes to
the backend — red without the fixture (1 failed), green with it (225
passed together with the whole computer-use file).
Running the suite could take over the machine it ran on. Two real
effects, both now closed:
- **The suite spoke through the speakers.** Once any test drove the
`voice.toggle` RPC with `action="tts"`, the handler set
`HERMES_VOICE_TTS=1` in the *live process environment*, and the flag
outlived that test. Every later test that drove a turn to completion
then fed its final response text to `hermes_cli.voice.speak_text` on a
background thread - real synthesis, real playback, no API key needed
(the default `edge` provider is keyless). A developer heard the fixture
string "partial answer complete" out loud. Because the flag is set from
inside the process, `scripts/run_tests.sh`'s `env -i` never protected
against this.
`tests/conftest.py` now blanks `HERMES_VOICE`/`HERMES_VOICE_TTS` per
test, and a new autouse `_audio_playback_guard` stubs `speak_text` and
its playback binding outright, so the speakers stay shut even inside the
test that sets the flag itself. `@pytest.mark.real_audio_playback` opts
out.
- **The suite launched Chrome.** `tests/tools/test_browser_supervisor.py`
spawned a real browser on any machine with Chrome on `PATH`. Its
docstring promised a `HERMES_E2E_BROWSER=1` gate that existed nowhere in
the code. That gate is now real, and the file is marked `integration`
so the default marker filter excludes it. `scripts/run_tests.sh`
forwards `HERMES_E2E_BROWSER` so the documented manual run still works.
Miscellanea
- `tests/test_audio_playback_guard.py`: regression cover for both defences,
driving the real `voice.toggle` handler rather than a stand-in.
Two tests fail deterministically on main depending only on which SQLite the
test interpreter links — nothing about the code under test. Both are green in
isolation and red in the suite / on an older library, the worst diagnostic
shape.
Root cause A — WAL is not always WAL. Hermes refuses journal_mode=WAL on
SQLite builds carrying the upstream WAL-reset corruption bug (3.7.0–3.51.2,
excluding backports 3.50.7 / 3.44.6) and falls back to DELETE. On such a
build NO -wal sidecar is ever created, so
test_wal_checkpoint_truncates_wal_file asserts on a file that cannot exist.
Invisible locally when the repo .venv and the Hermes managed runtime link
different versions (observed: .venv 3.50.4 → DELETE, runtime 3.53.1 → WAL),
so the same test passes for one interpreter and fails for the other.
- tests/conftest.py: add a `requires_wal` marker plus a
pytest_collection_modifyitems hook that skips such tests when the linked
library will fall back to DELETE. The skip reason names the actual
version so it is diagnosable rather than mysterious.
- pyproject.toml: register the marker.
- test_kanban_db_repair.py: mark the -wal-sidecar test.
Root cause B — process-global warn-once dedup. The WAL-fallback warning is
emitted at most once per (process, db_label). Any earlier test in
test_kanban_db.py that opens a kanban.db consumes that one-shot, so
test_connect_falls_back_to_delete_on_locking_protocol sees zero warnings and
fails — but only as part of the file, never alone.
- test_kanban_db.py: clear both dedup sets in the test that asserts on the
warning, with a comment explaining the isolation trap.
The gate deliberately does NOT import hermes_state. That module computes
DEFAULT_DB_PATH from get_hermes_home() at import time, so importing it during
collection — before the per-test _isolate_hermes_home fixture redirects
HERMES_HOME — permanently caches the developer's REAL ~/.hermes/state.db for
the whole session. The first version of this change did exactly that and made
tests read a live 31,881-session production database (test_console_engine
asserted "Total sessions: 2" and got 31881). The version predicate is
duplicated instead, and tests/test_conftest_wal_gate.py pins the two
implementations in agreement across every documented upstream boundary plus
guards against the import coming back.
Verified: on SQLite 3.50.4 the sidecar test SKIPS naming the version; on
3.53.1 it RUNS and passes, so coverage is not lost where WAL works. Clean
main fails exactly these 2 tests under
`scripts/run_tests.sh tests/hermes_cli/ tests/test_hermes_state.py`
(9926 passed, 2 failed); with this change the same scope is green.
Tests: 726 passed across the two kanban files, test_hermes_state.py, and the
new gate tests.
Follow-up to the #51899 pick, folding in the config-bridge half of the
competing PR #46925 (@bhanusharma, earliest submitter for the
ignored-channel gate):
- _apply_yaml_config: translate config.yaml slack.ignored_channels into
SLACK_IGNORED_CHANNELS (list or CSV), env-var-wins like every other
bridged Slack key.
- SlackAdapter._slack_ignored_channels / gateway.run's
_slack_ignored_channels_from_gateway_config: fall back to the
SLACK_IGNORED_CHANNELS env var when PlatformConfig.extra carries no
value, so top-level slack: blocks (which flow through the env bridge,
not extra) are honored at both the adapter and runner gates.
- conftest: force-clear SLACK_ALLOWED_CHANNELS / SLACK_IGNORED_CHANNELS /
SLACK_DISABLE_DMS between tests (config-loader side-effect leak class).
- Tests: env-bridge translation + precedence in test_config.py, env
fallback + extra-wins in test_slack_runner_ignored_channels.py.
Credit: #46925 by @bhanusharma proposed the same gate with the YAML→env
bridge; #51899 (picked as the base) carries the wider outbound/runner
coverage. Closes#46925 as consolidated here with first-submitter credit.
- Add SLACK_THREAD_REQUIRE_MENTION, SLACK_IGNORE_OTHER_USER_MENTIONS and
SLACK_REQUIRE_MENTION_CHANNELS to the conftest behavioral-env force-clear
list so config-loader side effects can't leak between tests (same class
of leak the existing SLACK_* entries guard against).
- Document thread_require_mention and require_mention_channels in the
Slack messaging guide next to the other mention-gating options.
* feat(attribution): conflict-free contributor mappings via contributors/emails/ directory
The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet:
every concurrent salvage PR appended entries to the same lines of the
same file, so parallel PRs re-conflicted on every merge to main.
New system: one file per email under contributors/emails/ — filename is
the commit-author email, first non-comment line is the GitHub login.
File additions never conflict, so any number of PRs can add mappings
concurrently.
- scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen)
merged with the directory at import time (directory wins). All
existing consumers (resolve_author, contributor_audit.py) unchanged.
- scripts/add_contributor.py: idempotent CLI to add a mapping; refuses
conflicting reassignments (incl. against the legacy map), validates
email/login shapes.
- contributor-check.yml: attribution gate now accepts a mapping file OR
a legacy entry; failure message prints the exact add_contributor
command. Also auto-resolves bare <login>@users.noreply.github.com
emails is intentionally NOT added (kept id+login form only, matching
previous behavior).
- contributor_audit.py: guidance now points at add_contributor.py.
- tests/scripts/test_contributor_map.py: 12 tests covering loader,
merge precedence, CLI idempotency/conflict/validation, subprocess E2E.
* feat(ci): one-shot per-file flake retry in the parallel test runner
A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry
counts as green but is loudly reported in a '⚠ FLAKY' summary section
(with both attempts' output preserved) so the flake gets fixed instead
of eating a full-run rerun. Deterministic failures fail both attempts —
regressions cannot be laundered green.
- --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables)
- E2E verified: simulated first-run-fail flake goes green with banner;
deterministic failure still exits 1; retries=0 restores old behavior.
This converts the dominant CI failure mode (one timing-sensitive test
flaking a 4600-test shard, requiring a manual 10-minute rerun and an
agent triage loop) into a self-healing retry that costs one file's
runtime.
* test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s
These guard against catastrophic regex backtracking (seconds-to-minutes
class), but 0.15s is within scheduler-stall noise on loaded shared CI
runners — test_max_accepted_separator_free_input_is_fast failed a CI
shard this week on runner load alone. 2.0s still catches the regression
class with zero flake surface.
* fix(ci): job timeouts everywhere + retries on all network installs
Reliability pass over every workflow:
- timeout-minutes on all 21 jobs that lacked one (a hung job previously
burned the 6-hour default runner budget)
- ./.github/actions/retry wrapped around every network-fetching install
that lacked it: pip installs (deploy-site, skills-index), npm ci
(deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker
test deps). Deterministic build steps (npm run build) deliberately
NOT retried — split into separate steps so a real build failure fails
fast instead of retrying 3x.
* docs(agents): document the file-retry flake policy
* fix(ci): curl retries on deploy hook + skills-index probe
* fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile
From the workflow reliability audit:
- tests.yml: duration-cache restore had NO restore-keys while saves use
run_id-suffixed keys — the cache never matched once, so LPT slicing
always ran blind and unbalanced slices pushed heavy files toward the
per-file timeout. One-line restore-keys fixes slice balancing.
- Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed):
'gh pr view || true' turned an API blip into 'label absent' → false
BLOCKING failure. Now 3x retry, and API failure is reported as an API
failure instead of a missing label.
- detect-changes action: compare API retried before failing open (was
silently running all lanes on any blip).
- uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried
so registry blips don't read as 'lockfile stale'.
- docker.yml merge job: imagetools create retried (Docker Hub eventual
consistency on just-pushed digests).
- Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to
curl --retry 3 (ADD cannot retry; checksums still enforced); npm
--fetch-retries=5; playwright chromium fetch retried 3x.
- Advisory artifact uploads (per-slice durations, ci-timings report)
get continue-on-error so an artifact-service blip can't fail a green
test slice.
* fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list
- test_tui_gateway_server.py: session.create / non-eager session.resume
arm a 50ms threading.Timer (_schedule_agent_build) that outlives its
test and fires into the NEXT test's _make_agent mock, racily
corrupting captured state (the recurring session_resume shard
failures). Replaced the per-test whack-a-mole stub with a module-wide
autouse fixture; the 3 worker-lifecycle tests that genuinely need the
deferred build opt back in via @pytest.mark.real_agent_prewarm (new
marker in pyproject).
- test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the
live PROVIDER_REGISTRY instead of a hand-list that had drifted
(missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto')
tests failed on any machine with HF_TOKEN exported. E2E-verified with
HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass.
* test: de-flake 30 timing-sensitive test files for loaded CI runners
Root-cause fixes from the flake audit (session-DB mining + repo sweep):
Event-based sync instead of sleep-sync:
- title_generator: mock sets threading.Event, wait(10) replaces
sleep(0.3) hoping the daemon thread got scheduled
- docker zombie_reaping / profile_gateway: poll-for-state helpers
replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async)
- process_registry tree test: select()-bounded readline replaces an
unbounded blocking read (parent wedge now fails THIS test with a clear
message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s
(the 1s partition window mid-interpreter-startup is how a child PID
escaped the live-system guard in CI)
Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors;
all of these complete in ms-to-1s when healthy so the raises cost
nothing on green runs):
- subprocess/thread waits <= 2s raised to 10-15s across mcp_tool,
mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe,
mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt,
voice_cli_integration, docker_environment, session_store_lock_io,
planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output
(joins now also assert not is_alive() so stragglers fail loudly)
- wall-clock discrimination ceilings loosened where the guarded hang is
10x larger: local_background_child_hang 4s->10s, interrupt_cleanup
setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup
5s->15s, protocol/gil-starvation fast-handler 0.5s->2s,
iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s
- narrow assertion windows widened: honcho first-turn wait 0.4..0.65 ->
0.25..2.0 (property is bounded-not-hung, not an exact wall-clock);
compression fork-lock TTL 1s->3s (12 refresh chances per lease);
compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0)
- telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under)
* fix(tests): repair indentation from de-flake batch edit
* fix(tests): harden env isolation and replace remaining sleep-sync races
The full 42k-test run and complete npm check surfaced three more classes:
- Environment isolation: local ~/.honcho defaultHost and SSH_* variables
leaked into Python/TUI tests. Pin the default Honcho host in the
hermetic fixture, isolate the one fallback test from ~/.honcho, and
blank SSH_* around terminalSetup tests. This flipped 20 false failures
back to deterministic behavior on developer machines.
- Background-thread sleep-sync: Honcho async writer tests patched
time.sleep globally, then busy-polled with that same mocked sleep. Under
full-suite load the poller could starve the writer. Each test now waits
on an Event emitted by the exact flush/retry transition; 30/30 passed
under 15-way contention.
- Desktop streaming: the test slept 80ms and assumed a 500ms timer could
not fire before its assertion. A loaded runner descheduled the test for
>500ms and both chunks arrived. Producer controls now gate second-chunk
and completion transitions explicitly.
Also make file-retry observability complete: a self-healed flaky file now
prints BOTH attempts' full output in the FLAKY summary. Two behavioral
runner tests prove pass-on-retry is green+loud+traceback-preserving, while
a deterministic failure remains red.
* refactor(ci): use gh bot pat, better retries
refactor(ci): use retry action for PR label fetch
the retry action now captures stdout as a step output, so it can serve
double duty: retry + output capture for commands like 'gh pr view' whose
result must be consumed by later steps.
Retry action gains:
- 'stdout' output (heredoc-delimited to preserve newlines)
- tee to temp file so stdout still streams to the job log
- step id 'retry' for output reference
Both lint.yml and supply-chain-audit.yml now use the retry action
directly with 'command: gh pr view ...' and read
steps.<id>.outputs.stdout.
ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth
Replace secrets.GITHUB_TOKEN and github.token with
secrets.AUTOFIX_BOT_PAT across all workflows and composite actions
that use the gh CLI or GitHub API. The PAT has consistent permissions
across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate
limit sharing with the default token, and is already used by
js-autofix.yml for the same reasons.
19 sites swapped across 9 files:
- lint.yml (3): label fetch, comment post/edit, comment update
- supply-chain-audit.yml (5): scan, critical comment, unbounded dep
comment, label fetch, mcp-catalog comment
- lockfile-diff.yml (1): PR comment post/update
- skills-index-freshness.yml (1): issue creation on degraded probe
- skills-index.yml (2): index build, trigger deploy workflow
- upload_to_pypi.yml (2): release view poll, release upload
- ci.yml (1): timings report
- deploy-site.yml (2): skills index crawl
- detect-changes/action.yml (1): compare API call
---------
Co-authored-by: ethernet <arilotter@gmail.com>
* test: deflake CI and dev-machine flaky tests in bulk
Fixes ten distinct flake sources found by mining recent CI failures and
running the full suite on a dev machine with real user state:
CI-observed races:
- tests/conftest.py live-system guard: allow signal 0 (pure liveness
probe) through _guarded_kill/_guarded_killpg. psutil.pid_exists()
probes a just-killed grandchild reparented to init; the subtree check
fails for it and the guard RuntimeError'd
test_entire_tree_is_sigkilled_not_just_parent intermittently on
unrelated PRs.
Hermeticity flakes (fail on dev machines with real state, pass on CI):
- agent/coding_context.py: _marker_root() now skips the shared temp
root (tempfile.gettempdir()) like it skips $HOME — a stray
/tmp/package.json flipped every tmp_path test into the coding
posture (9 failures in test_coding_context.py).
- test_agent_guardrails.py: pin MAX_CONCURRENT_CHILDREN=3 via autouse
monkeypatch instead of freezing the user's real config value at
import time (import-time vs call-time config mismatch).
- test_web_tools_config.py: TestCheckWebApiKey now neutralizes the
ddgs package probe and registry providers — the optional ddgs
package in a dev venv lit up the fallback backend.
- test_credential_pool.py: block claude_code/hermes-oauth credential
autodiscovery in the two pool-merge tests that assert exact id
lists (a real ~/.claude/.credentials.json seeded an extra entry).
- test_modal_sandbox_fixes.py: clear _permanent_approved /
_session_approved — the user's real command_allowlist silently
approved the guard-escalation commands under test.
- test_setup_irc.py: stub prompt_checklist to select only the IRC row;
the non-TTY cancel fallback re-ran the real configured platforms'
interactive setup_fn, which hit input() under captured stdin.
- test_doctor.py: TestGitHubTokenCheck now patches the module-level
HERMES_HOME constant (the file's established pattern) instead of
only setenv — doctor was running PRAGMA integrity_check against the
real multi-GB state.db and blowing the 300s per-file budget.
Latent atexit-duplication (same _enter_buffered_busy class as #34217):
- test_undo_command.py: drop importlib.reload(tui_gateway.server) in
fixture teardown; reload re-registers the module's atexit hooks.
- test_session_platform_resolution.py: drop per-test reload of
tui_gateway.server; every resolver reads env at call time.
* test: sentinel model value in ignore-user-config fallback assertion
With HERMES_IGNORE_USER_CONFIG=1, load_cli_config() falls back to the
repo-root cli-config.yaml (untracked, gitignored). On a dev machine that
file can legitimately set the same popular model the test hardcoded
(anthropic/claude-sonnet-4.6), flipping the != assertion locally while
CI (no cli-config.yaml) stayed green. Use an impossible sentinel model
name instead.
New hermes_cli/managed_scope.py resolves a system-level managed directory
(HERMES_MANAGED_DIR override > /etc/hermes), parses managed config.yaml/.env
with fail-open semantics, and exposes is_key_managed/is_env_managed helpers.
The system default is ignored under pytest and HERMES_MANAGED_DIR is added to
the conftest env scrub so a real managed scope can't leak into the suite.
Not wired into the load paths yet (Phases 2-3).
The pyproject addopts pin `--timeout-method=signal` relies on signal.SIGALRM,
which doesn't exist on Windows. pytest-timeout raised AttributeError at timer
setup and aborted the entire run before any test executed, so the suite was
unrunnable on Windows by default. Override timeout_method to "thread" on
Windows in pytest_configure; POSIX keeps the more reliable signal method.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends _live_system_guard in tests/conftest.py to block any subprocess
call that would run 'hermes update' (or 'python -m hermes_cli.main update')
against the real checkout.
These commands run git fetch origin + git pull, overwriting repo files
like pyproject.toml mid-test-run and corrupting every subsequent
subprocess that reads them. The spawned process uses setsid /
start_new_session=True so it's invisible to pytest's process tree
(PPid=1) — the corruption was essentially undetectable without
explicit inotify/SHA watchdogs.
Root cause of #43703 CI failures: tests in TestUpdateCommandPlatformGate
called _handle_update_command() with HERMES_MANAGED='' and no Popen mock,
causing the code to fall through and spawn a real 'hermes update --gateway'
that overwrote pyproject.toml with origin/main's content (which still
had '--timeout=30 --timeout-method=thread' in addopts while the PR had
already removed pytest-timeout).
The guard covers all three invocation patterns:
- 'hermes update' / 'hermes update --gateway' (direct or via setsid bash -c)
- 'python -m hermes_cli.main update --gateway'
- '.venv/bin/hermes update' (absolute path variant)
Does not false-positive on: git update-index, apt-get update,
pip install --upgrade, or any command lacking 'hermes'/'hermes_cli'.
Salvages 8 distinct fixes from a batch of PRs by @kyssta-exe, reapplied
onto current main (original branches were stale) with a few refinements.
- cron(jobs.py): load_jobs() validates top-level JSON shape — a bare
list auto-repairs into the {"jobs": [...]} dict; scalars/null raise a
clear RuntimeError instead of an uncaught AttributeError that took
down the whole cron subsystem (#37065, closes#36867).
- web(web_server.py): close the per-action log file handle after Popen
so the parent stops leaking one fd per spawned action (#36843).
- web(web_server.py): DELETE /api/env returns 400 for invalid key names
instead of a misleading 500, mirroring PUT /api/env (#36840).
- gateway(gateway.py): read /proc/<pid>/cmdline inside a with-block so
the fd is released immediately instead of relying on GC (#36804).
- web-tools(web_tools.py): include "xai" in check_web_api_key() so a
configured X.AI web backend reports as available (#36802).
- compression(conversation_compression.py): mark the feasibility check
done only after it completes, and default the gate to "not checked"
if the attribute is missing (#36803).
- completion(completion.py): replace `ls` with directory globbing in the
generated bash/zsh/fish profile listers — handles names with spaces
and skips non-directory entries (#36806).
- terminal-tool(terminal_tool.py): drop a duplicate `import threading`
(#36808).
- claw(claw.py): the migrate recommendation now points at the real
`hermes gateway stop` command instead of the non-existent
`hermes stop` (#36795, #36796, closes#36771).
- tests: guard against a leaked HERMES_CRON_SESSION breaking gateway
approval tests — add it to the hermetic conftest unset list (root
cause, protects every test) and pop it in the affected test's
setup_method (#36796).
Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Follow-up to the salvaged #30728:
- Gateway already exports _HERMES_GATEWAY=1 at startup (gateway/run.py) and
cli.py already keys off it. Drop the redundant new HERMES_IN_GATEWAY var;
guard stop/restart on _HERMES_GATEWAY instead. One marker for one fact.
- Drop the greedy \bgateway.*restart alternation from the cron lifecycle
filter — it false-positived on legit prompts that merely mention an
unrelated gateway + a restart (API/payment gateway monitoring). The
specific 'hermes gateway (restart|stop|start)' pattern already covers the
real command.
- Rework the two negative guard tests to sentinel the first downstream call
so they don't drive real signal delivery (tripped the live-system guard).
- Add false-positive regression cases to test_safe_commands.
Remove unused imports (F401) and duplicate/shadowed import
redefinitions (F811) across the codebase using ruff's safe
autofixes. No behavioral changes -- imports only.
- ~1400 safe autofixes applied across 644 files (net -1072 lines)
- __init__.py re-exports preserved (excluded from F401 removal so
public re-export surfaces stay intact)
- Re-exports that are imported or monkeypatched by tests but look
unused in their defining module are kept with explicit # noqa:
F401 (gateway/run.py load_dotenv; run_agent re-exports from
agent.message_sanitization, agent.context_compressor,
agent.retry_utils, agent.prompt_builder, agent.process_bootstrap,
agent.codex_responses_adapter)
- Unsafe F841 (unused-variable) fixes deliberately skipped -- those
can change behavior when the RHS has side effects
- ruff lints remain disabled in pyproject.toml (only PLW1514 is
selected); this is a one-time cleanup, not a config change
Verification:
- python -m compileall: clean
- pytest --collect-only: all 27161 tests collect (zero import errors)
- core entry points import clean (run_agent, model_tools, cli,
toolsets, hermes_state, batch_runner, gateway)
- static scan: every name any test imports directly from an edited
module still resolves
The cleanup-fix in the previous commit handles the graceful-exit leak: a
Hermes process that runs ``atexit`` will now actually wait on the docker
stop/rm worker thread, so containers either survive (persist mode) or are
fully removed (opt-out mode) by the time the interpreter exits.
But ``atexit`` doesn't fire on SIGKILL, OOM-kill, or terminal-window
close. Containers from those exits stay parked with no surviving Python
process to reuse or remove them, so they accumulate until the operator
intervenes with ``docker rm -f``. The cleanup-fix doesn't help this class
— there's no live cleanup() to fix.
This commit adds the safety net: a startup orphan reaper that runs once
per Hermes process and removes long-Exited hermes-labeled containers
that the prior commit couldn't reach.
Implementation:
* New ``reap_orphan_containers()`` in ``tools/environments/docker.py``.
Filters: ``label=hermes-agent=1`` + ``status=exited`` + (optional)
``label=hermes-profile=<current>``. Per-container ``docker inspect``
parses ``State.FinishedAt`` (with nanosecond-precision trimming for
Python's microsecond-bound ``fromisoformat``); containers older than
the threshold get ``docker rm -f``'d. The ``status=exited`` filter is
load-bearing — a running container may belong to a sibling Hermes
process whose reuse path will pick it up; killing it would crash the
sibling mid-command. Single-container failures are logged and the
sweep continues to the next candidate.
* New ``_maybe_reap_docker_orphans()`` helper in
``tools/terminal_tool.py``. Wired into ``_create_environment()`` for
``env_type == "docker"``. Gated by:
- ``terminal.docker_orphan_reaper: true`` (default; opt-out for
operators running multiple Hermes processes in the same profile
who don't trust the conservative defaults)
- ``_docker_orphan_reaper_ran`` module flag with double-checked
locking — parallel subagents and RL rollouts don't trigger N
concurrent docker ps storms
- Age threshold = ``2 × TERMINAL_LIFETIME_SECONDS`` with a 60s floor
(so ``TERMINAL_LIFETIME_SECONDS=0`` doesn't race the user's own
setup)
- Profile scoping — a research profile NEVER reaps the default
profile's stragglers
- Exception swallow — a janitor failure must never block container
creation
* New config ``terminal.docker_orphan_reaper`` wired through all four
config-bridge sites (cli.py, gateway/run.py, hermes_cli/config.py,
tests/conftest.py) and pinned by
``test_docker_orphan_reaper_is_bridged_everywhere``.
Coverage:
* 9 new unit tests in test_docker_environment.py — happy path, recent-
container sparing, profile scoping, unparseable-timestamp safety,
docker-ps-failure handling, partial-failure continuation, nanosecond
timestamp parsing, zero-value FinishedAt rejection.
* 6 new integration tests in test_docker_orphan_reaper_integration.py
— once-per-process gate, disable-flag respected, lifetime doubling
with 60s floor, current-profile filter wiring, exception swallow.
* 1 new bridge-invariant regression test.
Closes#20561 (combined with the two prior commits on this branch).
The Docker backend docs claim "Single persistent container — ONE long-
lived container shared across sessions, /new, /reset, and delegate_task
subagents. Stopped/removed on shutdown." In practice the code only
honored that contract within a single Python process via the in-memory
\`_active_environments[task_id]\` cache. Every \`hermes chat\` invocation
spawned a fresh \`hermes-<hex>\` container; older containers piled up in
\`Exited\` state and accumulated until manual \`docker rm\` (issue #20561).
Three root causes, all addressed by this commit:
1. No cross-process container discovery.
2. \`cleanup()\` used fire-and-forget \`subprocess.Popen("... &", shell=True)\`
which raced with parent-process exit — when Python exited promptly the
detached shell child got killed mid-\`docker stop\`, leaving stopped
containers behind.
3. The \`docker rm\` step in cleanup was gated on \`not self._persistent\`
(the bind-mount-persistence flag). Default config sets
\`container_persistent: true\`, so the default happy path skipped \`rm\`
entirely — even when the user explicitly didn't want cross-process
reuse, containers leaked.
Fix:
* Add \`DockerEnvironment.__init__(persist_across_processes=True)\`. When
true, init probes
\`docker ps -a --filter label=hermes-agent=1
--filter label=hermes-task-id=<task>
--filter label=hermes-profile=<profile>\`
and reuses a matching container (running → attach; stopped →
\`docker start\` → attach; \`docker start\` failure → fall through to a
fresh \`docker run\`). Multiple matches prefer the running one, with the
stragglers left for the orphan reaper (next commit) to clean up.
* Rewrite \`cleanup()\`. Uses \`subprocess.run(..., timeout=30)\` on a
daemon \`threading.Thread\`, not the racy \`Popen(... &)\`. The
\`_persistent\` guard is dropped on the \`rm\` step — \`rm\` now runs
whenever \`persist_across_processes\` is false, regardless of the
bind-mount-persistence setting. The leak class is gone in all
combinations.
* Add \`wait_for_cleanup(timeout)\`. \`tools/terminal_tool.py\`'s atexit
hook calls this on every active env, blocking up to 15s for the
cleanup thread before interpreter exit. Without this, \`hermes /quit\`
raced the daemon-thread teardown and dropped the stop/rm work.
* New config \`terminal.docker_persist_across_processes\` (default
\`true\` — restores the documented contract). Set \`false\` for hard
per-process isolation. Wired through all four config-bridge sites
(cli.py env_mappings, gateway/run.py _terminal_env_map,
hermes_cli/config.py _config_to_env_sync, tests/conftest.py env-strip
list); regression-pinned by
\`test_docker_persist_across_processes_is_bridged_everywhere\` matching
the existing pattern for docker_run_as_host_user / docker_env.
Reuse intentionally does NOT compare image / mounts / resources — only
the labels. Operators changing those settings should set
\`docker_persist_across_processes: false\` (or \`docker rm -f\` the
labeled container) to force a fresh start. This keeps the probe cheap
and the failure mode obvious.
Coverage: 12 new unit tests in tests/tools/test_docker_environment.py
covering reuse paths (running, stopped, fallback, opt-out, duplicate
preference) and cleanup behavior (persist-mode no-rm, opt-out always-rm,
no-Popen, wait_for_cleanup semantics, partial-init safety). Plus one
config-bridge regression pin.
Refs #20561
When these vars are set in the developer's shell, every /api/status call
triggers load_gateway_config() -> discover_plugins() -> the bundled
dashboard_auth/nous plugin auto-registers itself, leaking a provider into
the registry across tests on the same xdist worker. That breaks assertions
like 'auth_providers == []' (loopback) and '== ["stub"]' (gated) in
test_dashboard_auth_status_endpoint.py.
CI never has these set, so this only surfaced locally -- exactly the
hermeticity gap _hermetic_environment is meant to close. Add them to
_HERMES_BEHAVIORAL_VARS so the autouse fixture strips them, and to the
unset list in scripts/run_tests.sh as belt-and-suspenders for direct
pytest invocations.
* remove Vercel AI Gateway provider and Vercel Sandbox terminal backend
Both Vercel-hosted integrations are removed end-to-end. Users on the AI
Gateway should switch to OpenRouter or one of the other aggregators
(Nous Portal, Kilo Code). Users on the Vercel Sandbox backend should
switch to Docker, Modal, Daytona, or SSH.
What's removed:
- `plugins/model-providers/ai-gateway/` provider plugin
- `hermes_cli/vercel_auth.py` Vercel-Sandbox auth helper
- `tools/environments/vercel_sandbox.py` terminal backend
- `ai-gateway` provider wiring across auth, doctor, setup, models,
config, status, providers, main, web_server, model_normalize, dump
- `vercel_sandbox` backend wiring across terminal_tool, file_tools,
code_execution_tool, file_operations, approval, skills_tool,
environments/local, credential_files, lazy_deps, prompt_builder,
cli, gateway/run
- `AI_GATEWAY_BASE_URL` constant, `_AI_GATEWAY_HEADERS` auxiliary-client
header set, run_agent base-URL header/reasoning special-cases
- `[vercel]` pyproject extra and `vercel`/`vercel-workers` from uv.lock
- env vars: `AI_GATEWAY_API_KEY`, `AI_GATEWAY_BASE_URL`, `VERCEL_TOKEN`,
`VERCEL_PROJECT_ID`, `VERCEL_TEAM_ID`, `VERCEL_OIDC_TOKEN`,
`TERMINAL_VERCEL_RUNTIME`
- Tests: deletes test_ai_gateway_models.py and
test_vercel_sandbox_environment.py; scrubs references across 23
surviving test files (no entire tests deleted unless they were
dedicated to AI Gateway / Sandbox)
- Docs: provider tables, env-var reference, setup guides, security
notes, tool config, terminal-backend tables — English plus zh-Hans
i18n parity
- `hermes-agent` skill: provider table entry and remote-backend list
What stays (intentional):
- `popular-web-designs/templates/vercel.md` — CSS design reference,
unrelated to Vercel-the-AI-product
- `x-vercel-id` in `stream_diag.py` headers — generic Vercel CDN
response header, useful diag signal on any Vercel-hosted endpoint
- `vercel-labs/agent-browser` URL in browser config — lightpanda
browser project, different OSS effort
- `userStories.json` historical contributor entry mentioning Vercel
Sandbox — archive, not active docs
Validation:
- 1153 tests in the 22 targeted files pass (`scripts/run_tests.sh`)
- Full repo `py_compile` clean
- Live import of every touched module + invariant check (no
`ai-gateway` in `PROVIDER_REGISTRY`, no `_AI_GATEWAY_HEADERS`, no
`vercel_sandbox` in `_REMOTE_TERMINAL_BACKENDS`)
* test: convert profile-count check from change-detector to invariant
The hardcoded "== 34" assertion broke when ai-gateway was removed.
Per AGENTS.md change-detector-test guidance, assert the relationship
(registry count >= number of plugin dirs) instead of a literal count.
Counts shift when providers are added/removed; that's expected.
* ci(tests): install ripgrep from prebuilt tarball instead of apt
apt-get update + install of ripgrep takes ~4 min on the GHA Ubuntu
runners (the apt-get update against archive.ubuntu.com is the slow
part; ripgrep itself is small). Switching to the upstream musl
binary tarball cuts the step to a few seconds.
- Pinned to ripgrep 15.1.0 with sha256 verification (same hash as
published in the releases sha256 sidecar file).
- Drops the `rg` binary into /usr/local/bin so it is on PATH for
every subsequent step without GITHUB_PATH manipulation.
- Applied to both the test and e2e jobs in tests.yml.
* fix(cli): compile syntax check to tempdir, not source __pycache__
`_validate_critical_files_syntax` runs `py_compile.compile()` on each
critical bootstrap file after a successful `git pull`. The default
`py_compile` writes the resulting `.pyc` next to the source under
`__pycache__/`, which causes two real problems:
1. Parallel test workers walking the same source tree (e.g. running
the suite under per-file process isolation) can race against each
other on the `__pycache__` write — manifests as flaky 'directory
not empty' errors during teardown.
2. In production, the post-pull syntax check leaves a `.pyc` behind
that the next interpreter run might pick up — fine when the
interpreter version matches, sketchy if it doesn't.
Fix: write the compiled output to a `tempfile.TemporaryDirectory()`
that's discarded on function exit. We only care about the compile-or-not
signal, not the artifact.
* test(runner): per-file process isolation, drop manual state reset + xdist
Replace fragile manual _reset_module_state test fixtures with robust
per-file subprocess isolation. Each test file runs in a fresh
`python -m pytest <file>` subprocess via ThreadPoolExecutor. No xdist,
no custom pytest plugin, no shared worker state.
Key changes:
* scripts/run_tests_parallel.py — new runner: discovers test files,
runs N in parallel via ThreadPoolExecutor, captures stdout per file,
treats exit code 5 (no tests collected) as pass, kills all children
on exit. Change from cpu_count to cpu_count*2. The runner is
I/O-bound (waiting on subprocess.communicate() from pytest children)
The parent process does almost no CPU work, so 2x oversubscription
keeps more pipes full. When a file fails, immediately show the last
30 lines of pytest output (stack traces + FAILED summary) plus a
ready-to-copy repro command:
python -m pytest tests/agent/test_auxiliary_client.py
* scripts/run_tests.sh — delegates to run_tests_parallel.py
* .github/workflows/tests.yml — test step: python
scripts/run_tests_parallel.py
* pyproject.toml — drop pytest-xdist, pytest-split; simplify addopts
* tests/conftest.py — remove ~200 lines of manual state-reset fixtures
* AGENTS.md — update Testing section for per-file design
* test(runner): speed gateway test antipattern scan up
* fix(test): web search provider plugin test missing xai
* fix(tests): make 14 test files pass under per-file subprocess isolation
Tests that relied on cross-file state pollution from xdist workers
fail when run in isolation (per-file subprocess model). Root causes
and fixes:
Tool registry not populated:
- test_video_generation_tool_surface_matrix: add discover_builtin_tools()
- test_web_providers_brave_free/ddgs/searxng/general: autouse fixtures
registering all 8 bundled web providers, reset after each test
- test_website_policy: same provider registration pattern
- test_web_tools_tavily: same pattern across 3 dispatch test classes
- Also add is_safe_url/check_website_access mocks where SSRF check
blocks example.com (DNS resolution fails in isolated envs)
Stale check_fn cache:
- test_kanban_tools: invalidate_check_fn_cache() + _clear_tool_defs_cache()
in both kanban guidance tests (prior test cached False for kanban_show)
- test_discord_tool: cache invalidation in setup/teardown
- test_homeassistant_tool: invalidate_check_fn_cache() before registry queries
Module-level state pollution:
- test_auxiliary_client: autouse fixture clearing _aux_unhealthy_until cache
- test_skill_commands: set_session_vars() instead of patch.dict(os.environ)
(ContextVar takes precedence over os.environ)
- test_dm_topics: overwrite sys.modules + separate telegram.constants mock
+ force-reimport of gateway.platforms.telegram
- test_terminal_tool_requirements: removed duplicate class declaration,
autouse _clear_caches fixture
* change(tests): run_tests.sh explicitly includes env vars
instead of manually dropping some vars, now we just only include some
* fix(tests): 5 more isolation/NixOS fixes
- test_approval_plugin_hooks: isolate HERMES_HOME so real user's
command_allowlist doesn't short-circuit the approval path
- test_google_chat: skipif when Platform.GOOGLE_CHAT not in enum
(feature not merged on this branch)
- test_write_deny: test systemd prefix against tmp_path instead of
/etc/systemd which resolves to /nix/store on NixOS
- test_pty_bridge: use shutil.which('cat') instead of /bin/cat
(doesn't exist on NixOS)
- profiles.py: rmtree onexc handler chmod's parent dirs too, fixing
profile deletion when copytree preserved read-only modes from
nix store
* fix(tests): clear unhealthy cache in autouse fixture for auxiliary_client
* fix(tests): skip send_message when telegram not installed; handle missing worker_id in browser_supervisor
* fix: py3.11 rmtree onexc compat + belt-and-suspenders unhealthy cache clear for expired codex test
* fix: address PR #29016 review feedback
- Remove tracked .pytest-cache/ artifact and add to .gitignore
- Fix stale 'xdist worker' comment in conftest.py
- Deduplicate web provider registration into tests/tools/conftest.py
shared helper (register_all_web_providers), replacing 8 copy-pasted
blocks across 6 test files
- Update PR description: remove stale recovered-test-files claim,
fix worker count to match code (cpu_count*2)
* fix: eliminate race in stale-cache achievements test
The background scan thread could complete and overwrite _SNAPSHOT_CACHE
before evaluate_all() returned the stale data — only 10 fake sessions
made the scan finish instantly. Added scan_delay param to _FakeSessionDB
and set it to 2s in the stale-cache test so the background thread can't
win the race.
When Telegram topic mode is enabled, cron messages delivered to the bot's
root DM (TELEGRAM_HOME_CHANNEL without a thread id) land in the system
lobby — replies there are rebuffed with the lobby reminder and
reply_to_message_id is dropped, so users cannot interact with the cron
output (#24409).
Add an optional TELEGRAM_CRON_THREAD_ID env var that overrides
TELEGRAM_HOME_CHANNEL_THREAD_ID for cron deliveries only. Operators can
create a "Cron" forum topic in the DM, point this var at its thread id,
and replies to cron messages will land in that topic's existing session
instead of the lobby. The home-channel thread id (used elsewhere, e.g.
restart notifications) is unchanged, and explicit
deliver="telegram:chat:thread" targets continue to win over the env var.
Per the reporter's clarification on 2026-05-13, option (a) (cron-side
route to a dedicated topic + config knob) was chosen.
Fixes#24409
Salvages the substantive part of #22295 by @steezkelly. Adds the
missing HERMES_KANBAN_HOME, HERMES_KANBAN_RUN_ID, HERMES_KANBAN_CLAIM_LOCK,
HERMES_KANBAN_DISPATCH_IN_GATEWAY entries to _HERMES_BEHAVIORAL_VARS so
ambient developer-shell pins on those vars don't bleed into pytest runs.
The frozenset extraction + standalone regression test from the original
PR were dropped to keep the change minimal — main already maintains the
list inline.
The existing _live_system_guard (PR #23397) blocked os.kill / os.killpg
and a narrow subset of subprocess invocations. Tests still SIGTERMed the
live gateway today (May 10) because the guard had structural holes.
Plug them all:
- subprocess: also wrap getoutput, getstatusoutput
- os.system, os.popen - completely unwrapped before
- pty.spawn - completely unwrapped before
- asyncio.create_subprocess_exec / create_subprocess_shell - bypassed
the subprocess module entirely; now wrapped
- Subprocess command inspection now looks at the WHOLE command string,
not just tokens[0]. Catches sudo systemctl, env systemctl, bash -c
'systemctl', setsid systemctl, /usr/bin/systemctl, etc.
- New process-killer block: pkill / killall / taskkill / fuser
targeting hermes/python patterns is now refused
- os.kill PID 0 (own group) allowed; PID -1 (every process we can
signal) refused
- subprocess.Popen wrapper preserves __class_getitem__ so third-party
packages that use Popen[bytes] as a type annotation still import
Coverage is locked in by tests/test_live_system_guard_self_test.py -
exercises every primitive against a guaranteed-foreign PID and asserts
the guard fires. Adding a new kill primitive without updating the guard
breaks CI.
scripts/run_tests.sh now also force-loads ~/.hermes/pytest_live_guard.py
when present (developer-machine convenience), so even worktrees that
predate this commit get the protection on subsequent test runs through
the canonical wrapper.
The shutdown forensics added in #23285 caught tests/hermes_cli/ pytest
runs sending SIGTERM to the developer's live gateway 5+ times in 3
days. Root cause: when a single test forgets to mock os.kill or
find_gateway_pids, the real call leaks past the hermetic HERMES_HOME
isolation — find_gateway_pids' psutil scan walks the whole machine and
returns the live gateway PID, then the unmocked os.kill delivers the
signal.
Rather than audit and patch ~30 tests across cmd_update, kill_gateway_processes,
and stop_profile_gateway code paths, install a single autouse guard in
tests/conftest.py that blocks the two primitives that actually cause
the damage:
- os.kill rejects any PID outside the test process subtree with a
hard RuntimeError so the offending test gets a stack trace instead
of silently murdering the real gateway.
- subprocess.run / Popen / call / check_call / check_output reject
any 'systemctl <verb> hermes-gateway' invocation that would mutate
the live unit. Read-only systemctl calls (status, show, list-units)
still pass through.
We intentionally do NOT stub find_gateway_pids / _scan_gateway_pids —
tests of those functions themselves need the real implementation.
Discovery without delivery is harmless; the os.kill + systemctl guards
catch the actual damage path.
Tests that legitimately need real signal delivery (e.g. PTY tests
signalling their own child) opt out via
@pytest.mark.live_system_guard_bypass.
Validation: tests/hermes_cli/ + tests/cli/ + tests/gateway/ produce
the same 17 failures with and without this guard (all pre-existing on
main, unrelated to gateway-kill leaks). The live gateway survives the
test run that previously SIGTERMed it.
When the active main model has native vision and the provider supports
multimodal tool results (Anthropic, OpenAI Chat, Codex Responses, Gemini
3, OpenRouter, Nous), vision_analyze loads the image bytes and returns
them to the model as a multimodal tool-result envelope. The model then
sees the pixels directly on its next turn instead of receiving a lossy
text description from an auxiliary LLM.
Falls back to the legacy aux-LLM text path for non-vision models and
unverified providers.
Mirrors the architecture used in OpenCode, Claude Code, Codex CLI, and
Cline. All four converge on the same pattern: tool results carry image
content blocks for vision-capable provider/model combinations.
Changes
- tools/vision_tools.py: _vision_analyze_native fast path + provider
capability table (_supports_media_in_tool_results). Schema description
updated to reflect new behaviour.
- agent/codex_responses_adapter.py: function_call_output.output now
accepts the array form for multimodal tool results (was string-only).
Preflight validates input_text/input_image parts.
- agent/auxiliary_client.py: _RUNTIME_MAIN_PROVIDER/_MODEL globals so
tools see the live CLI/gateway override, not the stale config.yaml
default. set_runtime_main()/clear_runtime_main() helpers.
- run_agent.py: AIAgent.run_conversation calls set_runtime_main at turn
start so vision_analyze's fast-path check sees the actual runtime.
- tests/conftest.py: clear runtime-main override between tests.
Tests
- tests/tools/test_vision_native_fast_path.py: provider capability
table, envelope shape, fast-path gating (vision-capable model uses
fast path; non-vision model falls through to aux).
- tests/run_agent/test_codex_multimodal_tool_result.py: list tool
content becomes function_call_output.output array; preflight
preserves arrays and drops unknown part types.
Live verified
- Opus 4.6 + Sonnet 4.6 on OpenRouter: model calls vision_analyze on a
typed filepath, gets pixels back, reads exact text from images that
no aux description could capture (font color irony, multi-line
fruit-count list, etc.).
PR replaces the closed prior efforts (#16506 shipped the inbound user-
attached path; this PR closes the gap for tool-discovered images).
Two amplifying optimizations to per-turn overhead in the gateway:
1. get_tool_definitions() memoization (model_tools.py)
Keyed on (frozenset(enabled), frozenset(disabled),
registry._generation, config.yaml mtime+size). Only active when
quiet_mode=True (which is every hot-path caller — gateway,
AIAgent.__init__); quiet_mode=False keeps the existing print side
effects. Cached path returns a shallow-copy list sharing read-only
schema dicts.
Measured: 7.5 ms → 0.01 ms per call (~750× speedup). Gateway
constructs fresh AIAgent per message, so this saves ~7 ms/turn before
any LLM work.
2. check_fn() TTL cache (tools/registry.py)
check_fn callables like check_terminal_requirements probe external
state (Docker daemon, Modal SDK, playwright binary). For a long-lived
process, hitting them on every get_definitions() pass was pure waste
— external state changes on human timescales. 30 s TTL so env-var
flips (hermes tools enable X) propagate within a turn or two without
explicit invalidation.
Measured: first call 7.5ms → 1.6ms (check_fn probes now dominate);
subsequent calls ~0.01ms via the upstream memoization.
Invalidation surface:
- registry._generation bumps on register/deregister/register_toolset_alias,
invalidating the memoized definitions automatically.
- config.yaml mtime in the cache key captures user-visible config edits
affecting dynamic schemas (execute_code mode, discord allowlist).
- invalidate_check_fn_cache() exposed for explicit flushes (e.g. after
hermes tools enable/disable).
- tests/conftest.py autouse fixture clears both caches before every test
so env-var monkeypatches don't see stale results.
Also fixes a regression from PR #17046 that I missed:
- tools/web_tools.py — Firecrawl was removed from module scope by the
lazy import, breaking 8 tests that patch 'tools.web_tools.Firecrawl'.
Applied the same _FirecrawlProxy pattern used in auxiliary_client/
run_agent for OpenAI (module-level proxy that looks like the class
but imports the SDK on first call/isinstance; patch() replaces the
attribute as usual).
Verified:
- 49/49 tests/tools/test_web_tools_config.py pass (was 8 failing on main)
- 68/68 tests/tools/test_homeassistant_tool.py pass (was 1 failing in
the full suite due to check_fn TTL cross-test pollution; fixed by
the autouse fixture)
- 3887/3895 tests/tools/ (8 pre-existing fails: 2 delegate, 1 mcp
dynamic discovery, 5 mcp structured content — all confirmed on main)
- 2973/2976 tests/agent/ + tests/run_agent/ (3 pre-existing fails)
- 868/868 tests/run_agent/ (excluding test_run_agent.py which has
pre-existing suite-level issues)
- Live smoke: 2 turns + /model switch + tool calls, zero errors in
agent.log session window.
Co-authored-by: teknium1 <teknium@users.noreply.github.com>
load_gateway_config() has a side effect: when config.yaml contains
platform-gating keys (slack.require_mention, slack.strict_mention,
slack.free_response_channels, slack.allow_bots, slack.reactions, plus
analogous keys for discord/telegram/whatsapp/dingtalk/matrix), it calls
os.environ[KEY] = ... to bridge them to env-var form.
monkeypatch.delenv doesn't track direct os.environ mutations made
inside the test body, so tests that call load_gateway_config() leak
those env vars into later tests on the same xdist worker. The failure
mode is flaky seed-dependent: test_top_level_message_requires_mention_
even_with_session (and siblings in TestThreadReplyHandling) pass when
SLACK_REQUIRE_MENTION is unset but fail when a leaked value of 'false'
is present.
Add the gating env vars to _HERMES_BEHAVIORAL_VARS so the hermetic
autouse fixture blanks them on every test setup, closing the leak
regardless of which test sets them.
terry197913