OFFSET paging made the streaming export O(n^2) on huge transcripts;
after_id keyset paging keeps each page seek O(1). Adds after_id to
SessionDB.get_messages (ascending-only, guarded against latest/offset
combos).
The mid-setup CLI resume path loads only the tip session's rows, so
gate it with a tip-only count instead of the full-lineage count (which
over-rejected heavily-compressed sessions). Transient guard failures
(locked DB, adaptor stores) now log and proceed instead of blocking
resume with a new error.
sessions.max_resume_messages / sessions.max_export_messages (default
20000, 0 disables) replace the hardcoded hard-rejects, and the CLI
'sessions export' guard becomes per-session instead of cumulative so
full-DB backups of many small sessions keep working. Error guidance now
points at the config override instead of the (corruption-only) repair
command.
- Heartbeat tests: holder signals a threading.Event after acquiring
_SKILLS_PROFILE_LOCK; the scenario waits on it via run_in_executor
instead of sleeping 50ms and hoping.
- Fix the TestConfigMutationLock comment to describe the probabilistic
slow-save interleave the code actually implements.
- New regression test: PUT /api/dashboard/plugin-providers must hold
_CONFIG_MUTATION_LOCK — a concurrent locked writer survives.
Two follow-ups to the off-loop move, from external review (both verified,
the second larger than reported):
- Config read-modify-write handlers moved to worker threads could now
interleave — _CONFIG_LOCK covers each load/save individually, never the
span between them; the event loop used to serialize these accidentally.
New _CONFIG_MUTATION_LOCK (worker-threads only, so it can never block
the loop) held across the whole load→mutate→save span in all seven RMW
handlers. update_config_raw skipped: it's a full-document replace with
no server-side read, so a lock cannot close its client-side window.
- The review flagged two skills routes still taking _SKILLS_PROFILE_LOCK
on the event loop; a systematic audit of hermes_cli/web_routers/ found
24 on-loop routes (skills 5, mcp 9, tools 10, cron 1). All moved to the
same inner-_run + asyncio.to_thread pattern, mutating ones under the
mutation lock, uniform lock order (_SKILLS_PROFILE_LOCK →
_CONFIG_MUTATION_LOCK). Await-safe _config_profile_scope routes, plain
def routes, and already-threaded routes unchanged.
Regression tests: concurrent theme+font updates both survive (fails with
the lock nulled: "theme write lost to a concurrent font write"); event
loop stays responsive while the profile lock is held during GET
/api/skills. 214 tests passing across the touched suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The diagnostics loop watchdog caught GET /api/config freezing the gateway
event loop for >1s, stack-sampled blocking on _SKILLS_PROFILE_LOCK inside
_profile_scope. Any async handler that entered _profile_scope (process-wide
threading lock) or called load_config()/save_config() on-loop could stall
every chat and WebSocket at once while a slow lock-holder ran.
Move 28 such handlers to the existing inner-_run + asyncio.to_thread
pattern (contextvar-safe: the whole scope enter/body/exit stays inside one
worker thread). Handlers using the await-safe _config_profile_scope, plain
def endpoints (FastAPI threadpool), and tui_gateway's contextvar-only
decorator are unaffected and unchanged.
Regression test holds _SKILLS_PROFILE_LOCK in a thread while calling
GET /api/config and asserts an event-loop heartbeat keeps ticking; it fails
against the pre-fix code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the 'describe everything in thorough detail' auto image-preprocess
prompt with a concise 2-4 sentence summary prompt so image-bearing gateway
messages stop generating ~2000-char descriptions (35s+ on local models).
Prompt-only variant of #10852: the max_tokens=500 cap and the
preserve_max_tokens aux-client plumbing from the original PR are
intentionally dropped to stay compatible with the max-tokens-knob policy
direction (#75253 removes hardcoded vision caps).
Fixes#10809
An auth failure on the very first connect returned out of the run loop
instead of parking. That ended the run task, and the task is the only
listener on _reconnect_event — so the server stayed dead for the life of
the process. `hermes mcp login`, a /mcp refresh, and the 300s self-probe
all had nothing left to wake, and the only cure was a full restart.
_classify_mcp_failure already calls 401/403 "permanent" and documents
that run() parks those immediately; the early return above it meant auth
was the one permanent failure that never got there. Park it with the
others and keep the tailored log line, now pointing at `hermes mcp
login <server>`.
The windows-compat change-detector checked for the literal string
'start_new_session=True', but the systemd scope isolation path
conditionally uses start_new_session=False (the scope creates its own
session/cgroup). Assert 'start_new_session=' instead — the value may
now be a variable.
When Hermes runs as a systemd gateway with MemoryHigh/MemoryMax limits,
local background terminal commands (terminal(background=true)) inherit the
gateway's cgroup. A memory-heavy executor (Codex, tests, Node) can push
the whole cgroup past MemoryMax and trigger systemd-oomd to kill the
ENTIRE gateway — taking down the messaging control plane and silently
losing the active turn.
Root cause: tools/process_registry.py::spawn_local() uses
start_new_session=True (creates a process session/group, NOT a resource
cgroup). The spawned process tree stays in the gateway's systemd cgroup.
Fix: when running under a service manager (detected via the existing
is_gateway_supervisor_process() helper), wrap the pipe-mode spawn command
in 'systemd-run --user --scope --unit=hermes-worker-<id>' so the worker
gets its own transient cgroup. An OOM in the worker then kills only the
worker, not the gateway.
The systemd-run availability is probed once (a no-op /bin/true in a
transient scope) and cached, because the binary can exist on PATH while
the user D-Bus session is unavailable (system services, containers). If
unavailable, fall back to the current start_new_session=True behavior
with a debug log.
Scope: this covers the common background pipe-mode path. PTY mode
(PtyProcess.spawn) is left as future work — it uses a different spawn
mechanism and is used for interactive CLI tools where cgroup isolation
has additional considerations.
- Fix _usage_audit_path() to use _get_hermes_home() instead of hardcoded
Path.home() / '.hermes' (profile-safe resolution, sweeper finding)
- Rewrite skip_background_review tests to exercise finalize_turn() directly
instead of duplicating the guard expression (sweeper finding)
- Fix response_silent audit field to use _is_cron_silence_response()
instead of the buggy SILENT_MARKER substring check it was meant to
replace (simplify-code review finding)
- Remove dead 'model' in locals() guard — model is always in scope
before the try block (simplify-code review finding)
- Extract _stub_agent_for_finalize() helper to eliminate ~40 lines of
copy-pasted agent stubbing in tests (simplify-code review finding)
- Clean up 'Phase 0.5' instrumentation comments
Phase 0.5 of the Hermes Agent token leak mitigation plan: append a single
JSONL line to ~/.hermes/cron/usage_audit.jsonl after every cron LLM
invocation, capturing prompt/completion/total tokens, model, duration_ms,
deliver target, and error (when raised). Read from agent.session_*_tokens
which run_conversation already returns in its result dict.
Without this, we have no measured baseline to attribute token deltas to
subsequent mitigation phases. The plan's hard gate: observability lands
before any mitigation phase.
Writer NEVER raises — wrapped in a single try/except that logs a warning
on any json.dumps / mkdir / open failure so an audit-log bug cannot
break a cron job. Failure-path audit guard via locals() check covers
exceptions that fire before the fire_id is assigned.
No new dependencies, no new env vars (the plan rejected one in v2).
Tests: 7 new unit tests in tests/cron/test_usage_audit_logger.py covering
the success path, missing token info, swallowed writer exception, parent
dir creation, multiple appends, and unicode preservation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 8 of the Hermes Agent token leak mitigation plan
(ralplan-hermes-token-leaks.md §3.9). Adds a boolean kwarg
`skip_background_review` (default False) to AIAgent.__init__ that
suppresses the end-of-turn _spawn_background_review fork.
Each background review fork instantiates a new AIAgent with its own
~15K input tokens + up to 8 LLM iterations, accumulating ~30K tokens
per event in the worst case. On cron sessions there is no
human-in-the-loop benefit from the review (no skill-creation pressure,
nobody curating MEMORY.md), so the cost is pure waste.
The end-of-turn guard now reads:
if (final_response and not interrupted
and not getattr(self, "skip_background_review", False)
and (_should_review_memory or _should_review_skills)):
skip_memory=True already disables the memory-review trigger; this
flag is the explicit single-switch off for both review paths.
Defaults to False, so behavior is unchanged for gateway/CLI callers
that omit the kwarg.
Tests: 5 new unit tests in tests/agent/test_skip_background_review.py
covering the default value, flag persistence, the gate short-circuit,
the gate fall-through, and a source-text assertion that the cron
scheduler sets the flag to True (separate commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_fallback_session_info` returned `_default_session_cwd()` — the directory the
gateway process happened to start in — so a session resumed without a built
agent told its client the wrong workspace, and the desktop Files pane painted
the wrong project even after the renderer rebound correctly.
Return the session's own cwd and always emit `branch` ("" outside a git repo)
so a client can clear a stale label instead of retaining it. This matches the
contract `_lazy_session_info` already follows a few hundred lines above.
Co-authored-by: ZHJay <ZHJay@users.noreply.github.com>
The main-checkout test compared the two probe roots with raw string
equality. When they differed only in separator spelling, the repo's own
checkout was misclassified as a linked worktree: it fell through to the
worktree branch and was labeled by directory basename. The sidebar then
showed one checkout twice — a dir-labeled lane plus the branch-labeled
`main` lane built from the same sessions.
Compare with `_path_key` so platform path identity decides, matching how
every other path comparison in this module is already keyed.
Tests cover the single-checkout case and the main + linked-worktree case;
both fail before this change (the lane comes back labeled `repo`, not
`main`).
Fleet audit showed these task skills are commonly needed across users;
shipping bundled per Teknium's direction. Docs and tests follow the
bundled paths.
Surfaced during the post-merge review pass on our own #81113 follow-up:
cached_fetch_api_models gained _cache_entry_valid (numeric-'at'
validation) but its sibling cached_provider_model_ids still did
float(entry.get('at', 0)), which raises ValueError/TypeError on a
hand-edited or corrupted provider_models_cache.json row and propagates
uncaught into the /model picker call sites. Same fix, same helper:
corrupt rows are now a cache miss (live fetch), never an exception.
Both wrappers now share the identical validity predicate, closing the
divergence the 'mirrors' docstring promised away.
Also two test nits from the same review: unused OrderedDict import
dropped and the drain-order assertion strengthened to pin LRU-first
FIFO order in tests/gateway/test_agent_cache_pressure.py.
Mutation-checked: restoring the raising float() form makes the new
corrupt-at tests fail.
Rescope: hermes verify fills only the runtime-smoke gap and plugs into
the pieces Hermes already has instead of standing beside them.
- agent/verification_evidence.py: record_verify_run() — explicit ledger
write for hermes verify results (shared _insert_evidence factored out
of record_terminal_result). Passing runs mark the workspace passed
like scripts/run_tests.sh; failures are recorded; --phase/--skip-start
runs are recorded as targeted scope.
- hermes_cli/verify_cmd.py: record results into the ledger on completion
(fail-silent, HERMES_SESSION_ID attribution); on the detect path merge
detect_project_facts verify commands the recipe missed into the
recipe's test list (never applied to a saved manifest).
- agent/verification_stop.py: recipe-aware nudge — when the workspace
has a runnable recipe (start command or .hermes/environment.json),
suggest hermes verify --json as the preferred full check; cheap,
try/except-guarded detection that can never break the nudge path.
- agent/verify/recipes.py: document layer ownership (coding_context =
cheap prompt facts; verify/recipes = deep runtime recipe).
- tests/verify/test_ledger_and_nudge_integration.py: 17 tests covering
ledger pass/fail recording, the closed edit->nudge->verify->satisfied
loop, recipe-aware nudge wording + fail-silence, and the facts merge.
vision_analyze reads container-only images by exec-reading them inside the
sandbox, but unlike terminal_tool it never triggered environment creation. Under
a non-local backend (ssh, docker, ...), a session whose first action was
vision_analyze on a remote path failed with 'no active sandbox session' until an
unrelated terminal command happened to establish the connection.
Add terminal_tool.ensure_task_env(task_id), a public lazy get-or-create that
reuses the terminal tool's own creation machinery, and call it from
image_source._resolve_container_fallback before the in-sandbox read. Extract the
ssh/container config-dict builders so both paths derive settings identically
(no duplication). Best-effort and fail-closed: a failed bring-up leaves the
existing 'no active sandbox' error intact, never a host read.
Fixes#62825
The modal/browserbase test file replaces tools.environments.base with a
SimpleNamespace stub; terminal_tool now imports EnvironmentConnectionError
from that module, so the stub must provide it too.
Connection-class infrastructure failures on remote terminal backends (SSH
host unreachable/timed out, Docker daemon down or missing, remote file
sync failing on a dead link) previously surfaced to the model as raised
RuntimeError tracebacks. The model got a stack blob with no guidance and
the failure was indistinguishable from a tool bug.
Now:
- New EnvironmentConnectionError(RuntimeError) in tools/environments/base.py
carrying a reason + retry_hint. Subclassing RuntimeError keeps every
existing catcher working.
- ssh.py classifies connect-refused, connect-timeout, scp, remote mkdir,
bulk upload/download, and remote rm failures as connection errors.
- docker.py classifies all four _ensure_docker_available() failure paths
(missing exe, non-executable exe, daemon timeout, `docker version`
failure).
- terminal_tool catches EnvironmentConnectionError and returns a
structured tool result the model can act on:
{"status": "degraded", "reason": ..., "retry_hint": ..., "exit_code": -1}
The failed backend is evicted from the environment cache so a later
call retries from scratch — recovery is automatic once the backend is
reachable again.
- Config gate terminal.degraded_mode: warn|fail (default warn) in
config.yaml, bridged as TERMINAL_DEGRADED_MODE across all four bridge
sites (cli.py env_mappings, gateway/run.py _terminal_env_map,
TERMINAL_CONFIG_ENV_MAP, DEFAULT_CONFIG). "fail" preserves the
historical error+traceback tool result.
- Command failures (nonzero exit, command-not-found) are NOT touched —
only infrastructure failures classify as degraded.
Tests: tests/tools/test_terminal_degraded_mode.py (15 tests) covering
exception classification for ssh+docker, structured degraded results,
no-caching of degraded envs, recovery after the backend returns,
nonzero-exit results unaffected, fail-mode preservation, invalid-mode
fallback to warn, and the four-site config bridge invariant.
Inspired by: Claude Cowork degraded-backend behavior (idea-level,
docs-only evidence).
Adds a bounded, read-only health probe per CONFIGURED tool backend, run
only when the user explicitly passes `--live` (real network calls):
- Firecrawl: credit-usage metadata GET (auth check, no scrape spend)
- FAL: models metadata GET (never a generation)
- Browser: headless launch + about:blank + close (full cleanup)
- MCP: initialize + tools/list per configured server (reuses the
`hermes mcp test` machinery in mcp_config._probe_single_server)
- TTS/STT: provider models/voices list GET (openai/groq/elevenlabs);
local providers (edge/piper/faster-whisper/...) skipped
Invariants:
- Opt-in only: zero probes without --live (default False)
- Bounded: sequential, per-probe timeout (doctor.live_probe_timeout,
default 10s, config.yaml knob)
- Never mutates state; unconfigured backends skip with a note
- Failure isolation: every probe wrapped in a catch-all; a probe crash
can never break the doctor run; failures append to the issues summary
New: hermes_cli/doctor_live.py, tests/hermes_cli/test_doctor_live.py
(23 tests, probes mocked at the HTTP/client seam).
Wired: --live flag in subcommands/doctor.py; run_doctor calls
maybe_run_live_checks after all static checks.
Coordination: PR #70124 (--probe-routes) probes LLM routes; this flag
probes TOOL backends — different surface, no code-region collision
(the run_doctor hook here sits at the end-of-run summary, not the
API Connectivity section #70124 extends).
Inspired by: paradigmxyz/centaur tool-health-smoke (MIT/Apache-2.0);
sibling: #70124 (LLM route probes — different surface)
Per-task `output_schema` (JSON Schema object) on task items plus the
top-level single-goal form — a one-time static addition to the tool
schema (never varies per call).
- Child side: the schema is appended to the child's context as an
explicit OUTPUT CONTRACT block before spawn.
- Completion side: the parent validates the child's final answer with
jsonschema; on failure it sends exactly ONE bounded retry turn
carrying the validation errors verbatim (no schema re-paste).
- Result entries gain schema_valid (+ schema_retries, and schema_errors
on final failure) ONLY when a schema was requested; schema-less calls
keep a byte-identical result shape.
- Malformed schemas are rejected loudly at dispatch (coerce_output_schema
meta-validates via jsonschema's validator_for/check_schema).
- New helpers in tools/delegation_output_schema.py: coerce, contract
block, fence/prose-tolerant extraction+validation, retry message.
Pattern from: github/copilot-cli ctx.agent(prompt,{schema}) — PATTERN
ONLY, zero code/prompt text copied (proprietary); proven consumer:
delegate-task-output-patterns skill.
Tests: tests/tools/test_delegate_output_schema.py (24 tests — valid
first try, invalid->retry->valid, invalid twice -> schema_valid false +
errors surfaced, retry-exception degrade, no-schema legacy shape pin,
dispatch rejection, contract plumbing). Delegation suite: 221/221 green.
Add an optional `region: [x1, y1, x2, y2]` parameter to vision_analyze
(pixel coordinates in the ORIGINAL image space). The crop is applied
with Pillow BEFORE the downscale/embed-cap pipeline, so the cropped
region gets the full resolution budget — a zoom for reading small text
or UI details after a full shot.
- New `_crop_image_region` helper: clamps out-of-bounds coordinates to
the image, rejects zero-area/inverted/malformed regions with an error
naming the actual image dimensions so the model can retry sensibly.
- Wired into both the native fast path (`_vision_analyze_native`) and
the legacy aux-LLM path (`vision_analyze_tool`).
- Schema gains one static optional param (byte-stable thereafter); the
description documents the intended flow: full shot first, then zoom.
- No region supplied = behavior unchanged (regression-guarded).
Tests: tests/tools/test_vision_region.py (11 tests — crop applied,
clamping, zero-area rejection with dims, malformed input, pre-downscale
full-budget zoom, schema shape, handler pass-through, no-region
unchanged). Widened one narrow fake_native stub in test_vision_tools.py
to be kwargs-tolerant.
Ported from: QwenLM/qwen-code zoom-image.ts (Apache-2.0)
kshitij