OFFSET paging made the streaming export O(n^2) on huge transcripts;
after_id keyset paging keeps each page seek O(1). Adds after_id to
SessionDB.get_messages (ascending-only, guarded against latest/offset
combos).
The mid-setup CLI resume path loads only the tip session's rows, so
gate it with a tip-only count instead of the full-lineage count (which
over-rejected heavily-compressed sessions). Transient guard failures
(locked DB, adaptor stores) now log and proceed instead of blocking
resume with a new error.
The remote interceptor rebuilt session/messages requests from pathname
only, silently dropping limit/offset/order. Against a paginating remote
backend, getAllSessionMessages would refetch the same default page until
the safe-load guard threw, breaking export/artifacts/branch for remote
sessions over one page.
sessions.max_resume_messages / sessions.max_export_messages (default
20000, 0 disables) replace the hardcoded hard-rejects, and the CLI
'sessions export' guard becomes per-session instead of cumulative so
full-DB backups of many small sessions keep working. Error guidance now
points at the config override instead of the (corruption-only) repair
command.
Thread.start() failure is RuntimeError; catching BaseException here
swallowed KeyboardInterrupt/SystemExit without re-raise (unlike _worker,
which forwards them into the future).
- Heartbeat tests: holder signals a threading.Event after acquiring
_SKILLS_PROFILE_LOCK; the scenario waits on it via run_in_executor
instead of sleeping 50ms and hoping.
- Fix the TestConfigMutationLock comment to describe the probabilistic
slow-save interleave the code actually implements.
- New regression test: PUT /api/dashboard/plugin-providers must hold
_CONFIG_MUTATION_LOCK — a concurrent locked writer survives.
Two follow-ups to the off-loop move, from external review (both verified,
the second larger than reported):
- Config read-modify-write handlers moved to worker threads could now
interleave — _CONFIG_LOCK covers each load/save individually, never the
span between them; the event loop used to serialize these accidentally.
New _CONFIG_MUTATION_LOCK (worker-threads only, so it can never block
the loop) held across the whole load→mutate→save span in all seven RMW
handlers. update_config_raw skipped: it's a full-document replace with
no server-side read, so a lock cannot close its client-side window.
- The review flagged two skills routes still taking _SKILLS_PROFILE_LOCK
on the event loop; a systematic audit of hermes_cli/web_routers/ found
24 on-loop routes (skills 5, mcp 9, tools 10, cron 1). All moved to the
same inner-_run + asyncio.to_thread pattern, mutating ones under the
mutation lock, uniform lock order (_SKILLS_PROFILE_LOCK →
_CONFIG_MUTATION_LOCK). Await-safe _config_profile_scope routes, plain
def routes, and already-threaded routes unchanged.
Regression tests: concurrent theme+font updates both survive (fails with
the lock nulled: "theme write lost to a concurrent font write"); event
loop stays responsive while the profile lock is held during GET
/api/skills. 214 tests passing across the touched suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The diagnostics loop watchdog caught GET /api/config freezing the gateway
event loop for >1s, stack-sampled blocking on _SKILLS_PROFILE_LOCK inside
_profile_scope. Any async handler that entered _profile_scope (process-wide
threading lock) or called load_config()/save_config() on-loop could stall
every chat and WebSocket at once while a slow lock-holder ran.
Move 28 such handlers to the existing inner-_run + asyncio.to_thread
pattern (contextvar-safe: the whole scope enter/body/exit stays inside one
worker thread). Handlers using the await-safe _config_profile_scope, plain
def endpoints (FastAPI threadpool), and tui_gateway's contextvar-only
decorator are unaffected and unchanged.
Regression test holds _SKILLS_PROFILE_LOCK in a thread while calling
GET /api/config and asserts an event-loop heartbeat keeps ticking; it fails
against the pre-fix code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the 'describe everything in thorough detail' auto image-preprocess
prompt with a concise 2-4 sentence summary prompt so image-bearing gateway
messages stop generating ~2000-char descriptions (35s+ on local models).
Prompt-only variant of #10852: the max_tokens=500 cap and the
preserve_max_tokens aux-client plumbing from the original PR are
intentionally dropped to stay compatible with the max-tokens-knob policy
direction (#75253 removes hardcoded vision caps).
Fixes#10809
An auth failure on the very first connect returned out of the run loop
instead of parking. That ended the run task, and the task is the only
listener on _reconnect_event — so the server stayed dead for the life of
the process. `hermes mcp login`, a /mcp refresh, and the 300s self-probe
all had nothing left to wake, and the only cure was a full restart.
_classify_mcp_failure already calls 401/403 "permanent" and documents
that run() parks those immediately; the early return above it meant auth
was the one permanent failure that never got there. Park it with the
others and keep the tailored log line, now pointing at `hermes mcp
login <server>`.
The windows-compat change-detector checked for the literal string
'start_new_session=True', but the systemd scope isolation path
conditionally uses start_new_session=False (the scope creates its own
session/cgroup). Assert 'start_new_session=' instead — the value may
now be a variable.
The entry_data/lock_held path (used by mark_turn_active/clear_turn_active)
was eagerly constructing a full O(n) snapshot of all routing entries under
the lock on every turn, even when the DB upsert would succeed. Defer the
fallback_data construction to the except branch where it's actually needed,
and respect lock_held to avoid re-acquiring a non-reentrant _lock.
When Hermes runs as a systemd gateway with MemoryHigh/MemoryMax limits,
local background terminal commands (terminal(background=true)) inherit the
gateway's cgroup. A memory-heavy executor (Codex, tests, Node) can push
the whole cgroup past MemoryMax and trigger systemd-oomd to kill the
ENTIRE gateway — taking down the messaging control plane and silently
losing the active turn.
Root cause: tools/process_registry.py::spawn_local() uses
start_new_session=True (creates a process session/group, NOT a resource
cgroup). The spawned process tree stays in the gateway's systemd cgroup.
Fix: when running under a service manager (detected via the existing
is_gateway_supervisor_process() helper), wrap the pipe-mode spawn command
in 'systemd-run --user --scope --unit=hermes-worker-<id>' so the worker
gets its own transient cgroup. An OOM in the worker then kills only the
worker, not the gateway.
The systemd-run availability is probed once (a no-op /bin/true in a
transient scope) and cached, because the binary can exist on PATH while
the user D-Bus session is unavailable (system services, containers). If
unavailable, fall back to the current start_new_session=True behavior
with a debug log.
Scope: this covers the common background pipe-mode path. PTY mode
(PtyProcess.spawn) is left as future work — it uses a different spawn
mechanism and is used for interactive CLI tools where cgroup isolation
has additional considerations.
- Fix _usage_audit_path() to use _get_hermes_home() instead of hardcoded
Path.home() / '.hermes' (profile-safe resolution, sweeper finding)
- Rewrite skip_background_review tests to exercise finalize_turn() directly
instead of duplicating the guard expression (sweeper finding)
- Fix response_silent audit field to use _is_cron_silence_response()
instead of the buggy SILENT_MARKER substring check it was meant to
replace (simplify-code review finding)
- Remove dead 'model' in locals() guard — model is always in scope
before the try block (simplify-code review finding)
- Extract _stub_agent_for_finalize() helper to eliminate ~40 lines of
copy-pasted agent stubbing in tests (simplify-code review finding)
- Clean up 'Phase 0.5' instrumentation comments
Phase 0.5 of the Hermes Agent token leak mitigation plan: append a single
JSONL line to ~/.hermes/cron/usage_audit.jsonl after every cron LLM
invocation, capturing prompt/completion/total tokens, model, duration_ms,
deliver target, and error (when raised). Read from agent.session_*_tokens
which run_conversation already returns in its result dict.
Without this, we have no measured baseline to attribute token deltas to
subsequent mitigation phases. The plan's hard gate: observability lands
before any mitigation phase.
Writer NEVER raises — wrapped in a single try/except that logs a warning
on any json.dumps / mkdir / open failure so an audit-log bug cannot
break a cron job. Failure-path audit guard via locals() check covers
exceptions that fire before the fire_id is assigned.
No new dependencies, no new env vars (the plan rejected one in v2).
Tests: 7 new unit tests in tests/cron/test_usage_audit_logger.py covering
the success path, missing token info, swallowed writer exception, parent
dir creation, multiple appends, and unicode preservation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 8 wire-in + Vector 8 doc comment from
ralplan-hermes-token-leaks.md.
(1) Phase 8 wire-in: cron AIAgent construction now passes
skip_background_review=True. This suppresses the end-of-turn
skill/memory review fork (~30K tokens/event, ≤30K typical and
≤150K worst-case daily on bluenode) which has no human-in-the-loop
value for cron sessions.
(2) Vector 8 doc comment: a one-line comment immediately above the
AIAgent(...) construction documenting the verified-negative
finding that title generation does not run on the cron path
(maybe_auto_title is gateway/CLI-side only). Future contributors
won't accidentally introduce title-gen here without realizing it
would add ~600-1000 tokens/fire on a path that explicitly opts out
of memory/review/title overhead.
No new tests required for the doc comment (no behavior change). The
skip_background_review wiring is covered by the existing source-text
assertion in tests/agent/test_skip_background_review.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 8 of the Hermes Agent token leak mitigation plan
(ralplan-hermes-token-leaks.md §3.9). Adds a boolean kwarg
`skip_background_review` (default False) to AIAgent.__init__ that
suppresses the end-of-turn _spawn_background_review fork.
Each background review fork instantiates a new AIAgent with its own
~15K input tokens + up to 8 LLM iterations, accumulating ~30K tokens
per event in the worst case. On cron sessions there is no
human-in-the-loop benefit from the review (no skill-creation pressure,
nobody curating MEMORY.md), so the cost is pure waste.
The end-of-turn guard now reads:
if (final_response and not interrupted
and not getattr(self, "skip_background_review", False)
and (_should_review_memory or _should_review_skills)):
skip_memory=True already disables the memory-review trigger; this
flag is the explicit single-switch off for both review paths.
Defaults to False, so behavior is unchanged for gateway/CLI callers
that omit the kwarg.
Tests: 5 new unit tests in tests/agent/test_skip_background_review.py
covering the default value, flag persistence, the gate short-circuit,
the gate fall-through, and a source-text assertion that the cron
scheduler sets the flag to True (separate commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Table-driven across every session-scoped RPC that can hit a dead runtime
id, plus the invariants the scattered copies disagreed on: resume targets
the session-owning profile (never forks into the active one), the recovered
id publishes exactly once, drift during the resume aborts instead of
retrying, a resume that itself 404s rethrows the ORIGINAL error, and
recovery is bounded to a single retry.
These run without touching the REST layer because profile resolution is
injected — the pre-consolidation helper reached through resolveStoredSession
-> getSession(), which made its coverage depend on store state left behind
by whichever test file ran first (passed alone, timed out at 15s alongside
index.test.tsx).
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: rapsealk <rapsealk@users.noreply.github.com>
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
Same bug class, remaining call paths. runRewindSubmit handled only
"session busy" — a rewind runs right after cancelRun, and interrupting can
drop the gateway's session, so "Restore checkpoint" after a stop hit a dead
runtime id and surfaced the raw error. The session-tile delegate's
interrupt/submit had no recovery either, so a tile left open across sleep
was dead until reopened.
Both now use the shared resolver. The tile delegate resolves its durable id
by reversing the stored->runtime cache and repoints that mapping on
recovery, so later tile actions use the live binding instead of recovering
on every call.
Co-authored-by: JonthanaHanh <JonthanaHanh@users.noreply.github.com>
Co-authored-by: akivavh <akivavh@users.noreply.github.com>
After sleep/wake, a long idle, or a remote backend restart, the gateway
drops its in-memory runtime session while the desktop still holds the old
id. prompt.submit already recovered, so plain text kept working and the
failure looked selective: attaching an image or running /compress died
with a bare "session not found" and a new chat was the only workaround.
Attach runs BEFORE prompt.submit, so submit's recovery never got a
chance — image.attach_bytes / image.attach / file.attach failed first.
Route them (and session.compress) through the shared resolver, and thread
the recovered id back to the caller so the follow-up submit targets the
live session instead of the dead one.
Attachment bytes are read once, outside the retry, so recovering a large
upload doesn't re-read it. /compress deliberately does NOT opt into
timeout recovery: it is legitimately LLM-slow and retrying a timeout
would double a minutes-long call.
Reported-by: bapemonkey (Discord)
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: zzz163519 <zzz163519@users.noreply.github.com>
Co-authored-by: luxles <luxles@users.noreply.github.com>
main carried three hand-rolled copies of the same recovery policy —
prompt.submit (submit.ts), session.interrupt (cancelRun) and
session.redirect (steering) each open-coded
isSessionNotFoundError -> resolveSessionProfile -> session.resume ->
retry. Three copies of one policy is how call sites drift apart, and it
is why the RPCs that were never given a copy (attach, /compress,
checkpoint restore) still surface a raw "session not found" after
sleep/wake while plain text silently recovers.
Introduce withSessionNotFoundResume() as the single resolver and move
all three existing copies onto it. Profile resolution is injected rather
than imported so the helper is unit-testable without reaching through
resolveStoredSession -> getSession(); a drift callback lets each caller
keep its own abort semantics via SessionRecoveryAborted.
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Eight cases across the whole class, not just the reported path:
- cold resume rebinds from the selected row before resume settles
- an empty runtime cwd releases ownership (the permanent-staleness half)
- releasing leaves the PATH intact, so panes don't collapse and the persisted
workspace survives
- a session row outside the loaded sidebar page doesn't blank the pane
- a non-git workspace with a null git_repo_root still uses its row cwd
- the branch label clears so the previous project doesn't leak
Verified as a real barrier: reverting utils.ts fails 5 of these.
Co-authored-by: xxxigm <tuancanhnguyen706@gmail.com>
Co-authored-by: ZHJay <ZHJay@users.noreply.github.com>
Co-authored-by: worlldz <worlldz@users.noreply.github.com>
`_fallback_session_info` returned `_default_session_cwd()` — the directory the
gateway process happened to start in — so a session resumed without a built
agent told its client the wrong workspace, and the desktop Files pane painted
the wrong project even after the renderer rebound correctly.
Return the session's own cwd and always emit `branch` ("" outside a git repo)
so a client can clear a stale label instead of retaining it. This matches the
contract `_lazy_session_info` already follows a few hundred lines above.
Co-authored-by: ZHJay <ZHJay@users.noreply.github.com>
`session.info` claimed the cwd for whatever the selection happened to be, so a
background tile's payload could re-point a fresh draft at the tile's workspace.
Treat a nonempty `stored_session_id` as non-matching when no primary session is
selected; only an ABSENT id uses the selected-session fallback (the backend
omits it on a lazy session, and refusing there would leave the workspace
un-owned for the rest of the conversation).
Matching goes through the lineage rather than raw string equality: the backend
id is the live session_key, which auto-compression rotates to the continuation
tip, while a selection made from a pinned row holds the stable lineage root.
Comparing those literally reads one conversation as two.
Co-authored-by: ZHJay <ZHJay@users.noreply.github.com>
kshitij