Commit Graph

21127 Commits

Author SHA1 Message Date
kshitij 2ddd24ec1f fix: use is_job_runnable/effective_job_state in remaining pause-check sites
Two claim-failure diagnostic paths (cronjob_tools.py:629,921) still used
the old inline 'not enabled or state==paused' check. After get_job()
normalizes via effective_job_state, a half-paused record has
state='scheduled' and enabled=True, so the inline check returned False —
mislabeling the job as 'already being fired' instead of 'paused/disabled'.

Also hoists effective_job_state/is_job_runnable to the top-level import in
cronjob_tools.py (was function-local) and updates console_engine.py's
_format_job to use effective_job_state instead of the old inline
state-or-enabled derivation — a fourth display path the original PR missed.

Follow-up to PR #81287.
2026-08-08 13:48:00 +05:30
rjvandeve c7a5de7d6e fix(cron): make pause authoritative against half-paused records
pause_job already sets enabled=false atomically with state/paused_at, but
get_due_jobs only checked enabled — so a contradictory record
(enabled=true + paused_at/state=paused) still fired. That was the 07-30
outage failure mode: list looked frozen, fleet kept merging.

- is_job_runnable / effective_job_state: pause markers gate fire; display
  derives from the scheduler-honoured enabled flag so half-paused never
  renders as [paused]
- get_due_jobs self-heals enabled=false + logs error on contradiction
- claim_job_for_fire uses is_job_runnable (paused_at counts too)
- list/format paths use effective_job_state
- behavioural tests: pause blocks due fire; half-pause self-disables
2026-08-08 13:48:00 +05:30
hermes-seaeye[bot] 2d5e93161b fmt(js): npm run fix on merge (#81589)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-08-08 08:15:17 +00:00
kshitij a8ccd52123 refactor(sessions): accurate scope wording for tip-only resume rejections
SessionResumeTooLargeError said 'across its lineage' even when the CLI
mid-setup path counted only the tip segment; the exception now takes a
scope phrase.
2026-08-08 13:36:08 +05:30
kshitij edf2cb4bf8 perf(sessions): skip counting entirely when transcript guards are disabled
With sessions.max_*_messages: 0 the guards previously still ran an
unbounded COUNT (full lineage for resume) — the exact pathological work
disabling them is meant to avoid. Live callers use the raise side
effect only, so return 0 without touching the messages table.
2026-08-08 13:36:08 +05:30
kshitij ad59bd92c7 test(desktop): align renamed message-fetch mocks; brace query-param guards
The salvage's getLatestSessionMessages/getAllSessionMessages split left
two desktop test files mocking the old getSessionMessages name (vi.mock
partial-mock let the real function through, so calls hit the un-mocked
path). Also braces the new single-line if guards per the curly lint
rule.
2026-08-08 13:36:08 +05:30
kshitij e8b05dc6c2 perf(dashboard): keyset pagination for streaming session export
OFFSET paging made the streaming export O(n^2) on huge transcripts;
after_id keyset paging keeps each page seek O(1). Adds after_id to
SessionDB.get_messages (ascending-only, guarded against latest/offset
combos).
2026-08-08 13:36:08 +05:30
kshitij 5b4b9bbf77 fix(sessions): tip-only resume guard on the CLI mid-setup path; fail open on guard errors
The mid-setup CLI resume path loads only the tip session's rows, so
gate it with a tip-only count instead of the full-lineage count (which
over-rejected heavily-compressed sessions). Transient guard failures
(locked DB, adaptor stores) now log and proceed instead of blocking
resume with a new error.
2026-08-08 13:36:08 +05:30
kshitij 2607dc9a85 fix(desktop): forward pagination params through the remote session interceptor
The remote interceptor rebuilt session/messages requests from pathname
only, silently dropping limit/offset/order. Against a paginating remote
backend, getAllSessionMessages would refetch the same default page until
the safe-load guard threw, breaking export/artifacts/branch for remote
sessions over one page.
2026-08-08 13:36:08 +05:30
kshitij f0794640f6 feat(sessions): config-gate transcript safety limits
sessions.max_resume_messages / sessions.max_export_messages (default
20000, 0 disables) replace the hardcoded hard-rejects, and the CLI
'sessions export' guard becomes per-session instead of cumulative so
full-DB backups of many small sessions keep working. Error guidance now
points at the config override instead of the (corruption-only) repair
command.
2026-08-08 13:36:08 +05:30
kinsolee c750d5354a fix(sessions): prevent oversized transcripts from exhausting memory 2026-08-08 13:36:08 +05:30
kshitij 643910afe3 refactor(gateway): narrow worker-start guard to Exception
Thread.start() failure is RuntimeError; catching BaseException here
swallowed KeyboardInterrupt/SystemExit without re-raise (unlike _worker,
which forwards them into the future).
2026-08-08 13:31:04 +05:30
kshitij 0d312126a0 test(gateway): de-flake history-lookup timing tests; add worker-start-failure regression 2026-08-08 13:31:04 +05:30
kshitij 38cd1999cb fix(gateway): fail open + release admission slot when history-lookup worker cannot start 2026-08-08 13:31:04 +05:30
HenryG 271867f6fa fix(gateway): bound media history workers 2026-08-08 13:31:04 +05:30
HenryG e52acf76a1 fix(gateway): keep media history reads off event loop 2026-08-08 13:31:04 +05:30
kshitij c360333a3f test(dashboard): deterministic lock gating + plugin-providers RMW regression
- Heartbeat tests: holder signals a threading.Event after acquiring
  _SKILLS_PROFILE_LOCK; the scenario waits on it via run_in_executor
  instead of sleeping 50ms and hoping.
- Fix the TestConfigMutationLock comment to describe the probabilistic
  slow-save interleave the code actually implements.
- New regression test: PUT /api/dashboard/plugin-providers must hold
  _CONFIG_MUTATION_LOCK — a concurrent locked writer survives.
2026-08-08 13:28:28 +05:30
kshitij 4ecdee38a6 fix(dashboard): close config-RMW gaps left by the off-loop sweep 2026-08-08 13:28:28 +05:30
Royalaid 965a548788 fix(gateway): serialize config mutations and finish the router off-loop sweep
Two follow-ups to the off-loop move, from external review (both verified,
the second larger than reported):

- Config read-modify-write handlers moved to worker threads could now
  interleave — _CONFIG_LOCK covers each load/save individually, never the
  span between them; the event loop used to serialize these accidentally.
  New _CONFIG_MUTATION_LOCK (worker-threads only, so it can never block
  the loop) held across the whole load→mutate→save span in all seven RMW
  handlers. update_config_raw skipped: it's a full-document replace with
  no server-side read, so a lock cannot close its client-side window.

- The review flagged two skills routes still taking _SKILLS_PROFILE_LOCK
  on the event loop; a systematic audit of hermes_cli/web_routers/ found
  24 on-loop routes (skills 5, mcp 9, tools 10, cron 1). All moved to the
  same inner-_run + asyncio.to_thread pattern, mutating ones under the
  mutation lock, uniform lock order (_SKILLS_PROFILE_LOCK →
  _CONFIG_MUTATION_LOCK). Await-safe _config_profile_scope routes, plain
  def routes, and already-threaded routes unchanged.

Regression tests: concurrent theme+font updates both survive (fails with
the lock nulled: "theme write lost to a concurrent font write"); event
loop stays responsive while the profile lock is held during GET
/api/skills. 214 tests passing across the touched suites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 13:28:28 +05:30
Royalaid 52c9aee3bf fix(gateway): move _profile_scope and config I/O off the event loop in async handlers
The diagnostics loop watchdog caught GET /api/config freezing the gateway
event loop for >1s, stack-sampled blocking on _SKILLS_PROFILE_LOCK inside
_profile_scope. Any async handler that entered _profile_scope (process-wide
threading lock) or called load_config()/save_config() on-loop could stall
every chat and WebSocket at once while a slow lock-holder ran.

Move 28 such handlers to the existing inner-_run + asyncio.to_thread
pattern (contextvar-safe: the whole scope enter/body/exit stays inside one
worker thread). Handlers using the await-safe _config_profile_scope, plain
def endpoints (FastAPI threadpool), and tui_gateway's contextvar-only
decorator are unaffected and unchanged.

Regression test holds _SKILLS_PROFILE_LOCK in a thread while calling
GET /api/config and asserts an event-loop heartbeat keeps ticking; it fails
against the pre-fix code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 13:28:28 +05:30
gnanirahulnutakki e6b168855b fix(gateway): keep auto vision preprocess concise
Replace the 'describe everything in thorough detail' auto image-preprocess
prompt with a concise 2-4 sentence summary prompt so image-bearing gateway
messages stop generating ~2000-char descriptions (35s+ on local models).

Prompt-only variant of #10852: the max_tokens=500 cap and the
preserve_max_tokens aux-client plumbing from the original PR are
intentionally dropped to stay compatible with the max-tokens-knob policy
direction (#75253 removes hardcoded vision caps).

Fixes #10809
2026-08-08 13:28:19 +05:30
brooklyn! d92bfa0a38 Merge pull request #81570 from NousResearch/bb/figma-mcp-401
Recover an MCP server that 401s at startup instead of needing a restart
2026-08-08 02:24:59 -05:00
Brooklyn Nicholson f99d291247 fix(mcp): let a server that 401s at startup come back after re-login
An auth failure on the very first connect returned out of the run loop
instead of parking. That ended the run task, and the task is the only
listener on _reconnect_event — so the server stayed dead for the life of
the process. `hermes mcp login`, a /mcp refresh, and the 300s self-probe
all had nothing left to wake, and the only cure was a full restart.

_classify_mcp_failure already calls 401/403 "permanent" and documents
that run() parks those immediately; the early return above it meant auth
was the one permanent failure that never got there. Park it with the
others and keep the tailored log line, now pointing at `hermes mcp
login <server>`.
2026-08-08 02:21:36 -05:00
brooklyn! b9aa9289a8 Merge pull request #81414 from helix4u/fix/ssh-remote-tilde-cwd
fix(terminal): preserve SSH remote home cwd
2026-08-08 00:32:45 -05:00
Gille 9c69d98864 fix(terminal): preserve SSH remote home cwd 2026-08-07 18:41:57 -06:00
rob-maron b3aa561faf add Hermes headers to Fireworks provider (#81321) 2026-08-07 20:56:29 +00:00
kshitij a8c50eb1d8 fix: relax start_new_session assertion for systemd scope path
The windows-compat change-detector checked for the literal string
'start_new_session=True', but the systemd scope isolation path
conditionally uses start_new_session=False (the scope creates its own
session/cgroup). Assert 'start_new_session=' instead — the value may
now be a variable.
2026-08-08 01:12:16 +05:30
kshitij b5c2116783 fix: defer O(n) fallback_data construction to failure path in _save_entry
The entry_data/lock_held path (used by mark_turn_active/clear_turn_active)
was eagerly constructing a full O(n) snapshot of all routing entries under
the lock on every turn, even when the DB upsert would succeed.  Defer the
fallback_data construction to the except branch where it's actually needed,
and respect lock_held to avoid re-acquiring a non-reentrant _lock.
2026-08-08 01:12:16 +05:30
Dominic Bejar c5e032c804 fix(gateway): close ambiguous recovery cleanup gaps 2026-08-08 01:12:16 +05:30
Dominic Bejar 46b5314229 fix(terminal): harden scope fallback and memory override 2026-08-08 01:12:16 +05:30
Dominic Bejar 5ff328cc76 fix(gateway): make active turn markers failure-atomic 2026-08-08 01:12:16 +05:30
Dominic Bejar b0346ba42a fix(terminal): align worker limit with local guard 2026-08-08 01:12:16 +05:30
Dominic Bejar 5f93083221 fix(terminal): bound isolated worker memory 2026-08-08 01:12:16 +05:30
Dominic Bejar 0690fd77c6 fix(terminal): make systemd cleanup gateway-safe 2026-08-08 01:12:16 +05:30
Dominic Bejar 69397937dd fix(terminal): serialize systemd scope capability probe 2026-08-08 01:12:16 +05:30
Dominic Bejar 59a128c6fb fix(gateway): harden active turn marker lifecycle 2026-08-08 01:12:16 +05:30
Dominic Bejar 6774760b6f fix(gateway): recover exact turns after unclean exits 2026-08-08 01:12:16 +05:30
toprakeker 21de22a4ec fix(terminal): fully-qualified .scope unit name, exit-code check, already_exited cleanup (#70716) 2026-08-08 01:12:16 +05:30
toprakeker 7cfa90d90a fix(terminal): address review gaps — PTY isolation, unit-name kill, --quiet (#70716) 2026-08-08 01:12:16 +05:30
toprakeker 099eb73731 fix(terminal): isolate local background executors in their own systemd cgroup (#70716)
When Hermes runs as a systemd gateway with MemoryHigh/MemoryMax limits,
local background terminal commands (terminal(background=true)) inherit the
gateway's cgroup. A memory-heavy executor (Codex, tests, Node) can push
the whole cgroup past MemoryMax and trigger systemd-oomd to kill the
ENTIRE gateway — taking down the messaging control plane and silently
losing the active turn.

Root cause: tools/process_registry.py::spawn_local() uses
start_new_session=True (creates a process session/group, NOT a resource
cgroup). The spawned process tree stays in the gateway's systemd cgroup.

Fix: when running under a service manager (detected via the existing
is_gateway_supervisor_process() helper), wrap the pipe-mode spawn command
in 'systemd-run --user --scope --unit=hermes-worker-<id>' so the worker
gets its own transient cgroup. An OOM in the worker then kills only the
worker, not the gateway.

The systemd-run availability is probed once (a no-op /bin/true in a
transient scope) and cached, because the binary can exist on PATH while
the user D-Bus session is unavailable (system services, containers). If
unavailable, fall back to the current start_new_session=True behavior
with a debug log.

Scope: this covers the common background pipe-mode path. PTY mode
(PtyProcess.spawn) is left as future work — it uses a different spawn
mechanism and is used for interactive CLI tools where cgroup isolation
has additional considerations.
2026-08-08 01:12:16 +05:30
hermes-seaeye[bot] 005421d888 fmt(js): npm run fix on merge (#81276)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-08-07 19:02:38 +00:00
brooklyn! 33edfe9a81 Merge pull request #81261 from NousResearch/bb/desktop-stale-session-recovery-consolidation
fix(desktop): recover every session-scoped RPC after a stale runtime-session drop
2026-08-07 12:54:47 -06:00
kshitij 7307f88993 fix: follow-up for salvaged PR #18255
- Fix _usage_audit_path() to use _get_hermes_home() instead of hardcoded
  Path.home() / '.hermes' (profile-safe resolution, sweeper finding)
- Rewrite skip_background_review tests to exercise finalize_turn() directly
  instead of duplicating the guard expression (sweeper finding)
- Fix response_silent audit field to use _is_cron_silence_response()
  instead of the buggy SILENT_MARKER substring check it was meant to
  replace (simplify-code review finding)
- Remove dead 'model' in locals() guard — model is always in scope
  before the try block (simplify-code review finding)
- Extract _stub_agent_for_finalize() helper to eliminate ~40 lines of
  copy-pasted agent stubbing in tests (simplify-code review finding)
- Clean up 'Phase 0.5' instrumentation comments
2026-08-08 00:07:14 +05:30
0xarkstar 15927c1d24 feat(cron): add usage_audit.jsonl logger for cron token leak instrumentation
Phase 0.5 of the Hermes Agent token leak mitigation plan: append a single
JSONL line to ~/.hermes/cron/usage_audit.jsonl after every cron LLM
invocation, capturing prompt/completion/total tokens, model, duration_ms,
deliver target, and error (when raised). Read from agent.session_*_tokens
which run_conversation already returns in its result dict.

Without this, we have no measured baseline to attribute token deltas to
subsequent mitigation phases. The plan's hard gate: observability lands
before any mitigation phase.

Writer NEVER raises — wrapped in a single try/except that logs a warning
on any json.dumps / mkdir / open failure so an audit-log bug cannot
break a cron job. Failure-path audit guard via locals() check covers
exceptions that fire before the fire_id is assigned.

No new dependencies, no new env vars (the plan rejected one in v2).

Tests: 7 new unit tests in tests/cron/test_usage_audit_logger.py covering
the success path, missing token info, swallowed writer exception, parent
dir creation, multiple appends, and unicode preservation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-08-08 00:07:14 +05:30
0xarkstar d3e3c62344 feat(cron): set skip_background_review=True; doc title-generation non-presence
Phase 8 wire-in + Vector 8 doc comment from
ralplan-hermes-token-leaks.md.

(1) Phase 8 wire-in: cron AIAgent construction now passes
    skip_background_review=True. This suppresses the end-of-turn
    skill/memory review fork (~30K tokens/event, ≤30K typical and
    ≤150K worst-case daily on bluenode) which has no human-in-the-loop
    value for cron sessions.

(2) Vector 8 doc comment: a one-line comment immediately above the
    AIAgent(...) construction documenting the verified-negative
    finding that title generation does not run on the cron path
    (maybe_auto_title is gateway/CLI-side only). Future contributors
    won't accidentally introduce title-gen here without realizing it
    would add ~600-1000 tokens/fire on a path that explicitly opts out
    of memory/review/title overhead.

No new tests required for the doc comment (no behavior change). The
skip_background_review wiring is covered by the existing source-text
assertion in tests/agent/test_skip_background_review.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-08-08 00:07:14 +05:30
0xarkstar eaeba6474f feat(agent): add skip_background_review flag to AIAgent constructor
Phase 8 of the Hermes Agent token leak mitigation plan
(ralplan-hermes-token-leaks.md §3.9). Adds a boolean kwarg
`skip_background_review` (default False) to AIAgent.__init__ that
suppresses the end-of-turn _spawn_background_review fork.

Each background review fork instantiates a new AIAgent with its own
~15K input tokens + up to 8 LLM iterations, accumulating ~30K tokens
per event in the worst case. On cron sessions there is no
human-in-the-loop benefit from the review (no skill-creation pressure,
nobody curating MEMORY.md), so the cost is pure waste.

The end-of-turn guard now reads:

    if (final_response and not interrupted
            and not getattr(self, "skip_background_review", False)
            and (_should_review_memory or _should_review_skills)):

skip_memory=True already disables the memory-review trigger; this
flag is the explicit single-switch off for both review paths.

Defaults to False, so behavior is unchanged for gateway/CLI callers
that omit the kwarg.

Tests: 5 new unit tests in tests/agent/test_skip_background_review.py
covering the default value, flag persistence, the gate short-circuit,
the gate fall-through, and a source-text assertion that the cron
scheduler sets the flag to True (separate commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-08-08 00:07:14 +05:30
hermes-seaeye[bot] 8370141f1c fmt(js): npm run fix on merge (#81259)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-08-07 18:27:48 +00:00
bb 37aaecf48b test(desktop): cover the stale-session recovery bug class
Table-driven across every session-scoped RPC that can hit a dead runtime
id, plus the invariants the scattered copies disagreed on: resume targets
the session-owning profile (never forks into the active one), the recovered
id publishes exactly once, drift during the resume aborts instead of
retrying, a resume that itself 404s rethrows the ORIGINAL error, and
recovery is bounded to a single retry.

These run without touching the REST layer because profile resolution is
injected — the pre-consolidation helper reached through resolveStoredSession
-> getSession(), which made its coverage depend on store state left behind
by whichever test file ran first (passed alone, timed out at 15s alongside
index.test.tsx).

Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: rapsealk <rapsealk@users.noreply.github.com>
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
2026-08-07 13:24:10 -05:00
bb c1305b645d fix(desktop): recover checkpoint restore and tile actions after a stale session drop
Same bug class, remaining call paths. runRewindSubmit handled only
"session busy" — a rewind runs right after cancelRun, and interrupting can
drop the gateway's session, so "Restore checkpoint" after a stop hit a dead
runtime id and surfaced the raw error. The session-tile delegate's
interrupt/submit had no recovery either, so a tile left open across sleep
was dead until reopened.

Both now use the shared resolver. The tile delegate resolves its durable id
by reversing the stored->runtime cache and repoints that mapping on
recovery, so later tile actions use the live binding instead of recovering
on every call.

Co-authored-by: JonthanaHanh <JonthanaHanh@users.noreply.github.com>
Co-authored-by: akivavh <akivavh@users.noreply.github.com>
2026-08-07 13:24:10 -05:00
bb 3c4f5c5217 fix(desktop): recover image/file attach and /compress after a stale session drop
After sleep/wake, a long idle, or a remote backend restart, the gateway
drops its in-memory runtime session while the desktop still holds the old
id. prompt.submit already recovered, so plain text kept working and the
failure looked selective: attaching an image or running /compress died
with a bare "session not found" and a new chat was the only workaround.

Attach runs BEFORE prompt.submit, so submit's recovery never got a
chance — image.attach_bytes / image.attach / file.attach failed first.
Route them (and session.compress) through the shared resolver, and thread
the recovered id back to the caller so the follow-up submit targets the
live session instead of the dead one.

Attachment bytes are read once, outside the retry, so recovering a large
upload doesn't re-read it. /compress deliberately does NOT opt into
timeout recovery: it is legitimately LLM-slow and retrying a timeout
would double a minutes-long call.

Reported-by: bapemonkey (Discord)
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: zzz163519 <zzz163519@users.noreply.github.com>
Co-authored-by: luxles <luxles@users.noreply.github.com>
2026-08-07 13:24:09 -05:00