Commit Graph

3662 Commits

Author SHA1 Message Date
ethernet 6e67841a9a refactor(goals): share one dropped-write warning across managers
Review fixups for #88965. The goal, loop, and heartbeat managers each
had a copy of the same WARNING text. The shared _warn_dropped_write
helper in goals.py keeps the three logs identical and greppable as one
bug class. The _warm_goals_session_db parameter is now label. The old
name ctx said context, but the value is a log label.
2026-08-18 16:08:29 -07:00
ethernet 46d8cf0be3 fix(gateway): /loop paths warm the SessionDB cache off-loop
The loops delegation (previous commit) moved /loop onto the shared
bootstrap windows, but the loop-class gateway callers still constructed
LoopManager on the loop thread with no warm-up — the same false-ack
class as /goal, one sibling over:

- The /loop command handler: a cold init past the window made
  save_loop discard the write while the reply claimed the loop was
  set. Reproduced with a 2s init: "↻ Loop set" with nothing persisted;
  with the warm-up the loop persists.
- _post_turn_loop_completion: a cold cache at the turn boundary
  stalled the loop for the init duration and could drop the
  tick-completion write.
- _loop_wakeup_watcher: the scan reads every persisted loop, so a cold
  cache ran the state.db init on the loop thread before the first
  read.

All three now warm the cache off-loop first (same helper, same reason
as the goal paths). save_loop also logs at WARNING when it drops a
write, matching save_goal: the reply has already told the user the
loop was set.

Review findings (PR #88965): the goal and heartbeat command paths
warmed off-loop, the loop-class paths did not.
2026-08-18 16:08:29 -07:00
ethernet 246477a809 fix(gateway): /goal no longer lies when state.db init is slow
A fresh state.db init (schema DDL, FTS tables, first config import)
measures ~300ms warm on a fast machine. The gateway constructs
GoalManager on the event-loop thread, and a cold cache ran that init
behind a 0.25s bootstrap grace window: on a slow CI box the /goal set
path's waits expired and save_goal silently no-oped — the reply said
"Goal set (7-turn budget)..." but nothing persisted, and a fresh
GoalManager read back no state (first assertion passes, second fails).

Two changes, one per caller shape:

- Async callers (_get_goal_manager_for_event,
  _get_heartbeat_manager_for_event, _post_turn_goal_continuation, and
  the heartbeat poller) warm the SessionDB cache off-loop through the
  context-preserving executor before constructing the manager (shared
  _warm_goals_session_db helper). The loop never blocks and the first
  write lands at any init duration. A bare to_thread would lose the
  per-turn profile home override under multiplex; the executor hop
  keeps it (same pattern as the goal judge path).
- Sync callers (heartbeat persistence, _goal_still_active_for_session)
  cannot await, so the bootstrap windows stay: the call that starts the
  bootstrap waits a one-time init window (1.5s) instead of the short
  per-call window (0.25s), giving healthy cold inits room to land while
  a contended migration still degrades to None with only a bounded
  one-time stall. The bootstrap thread binds the caller's home as a
  contextvar override so a multiplexed worker cannot cache the default
  profile's DB under another profile's key.

save_goal and heartbeat save_state now log at WARNING when they drop a
write, because the reply has already told the user the state was set.
Regression test pins the contract: init past the window, write
persists, loop gap under 2s (the flake-policy floor for wall-clock
bounds; the slow-init margin grew to match, so the test still tells
on-loop from off-loop).

Independent diagnosis + measurement by jackulau (#88965 review); the
off-loop warm-up shape follows their harness table. Simplify-code
review (4-agent) contributed the helper extraction and the poller
warm-up.
2026-08-18 16:08:29 -07:00
Teknium c91681c69a fix(tools): large tool results persist to HERMES_HOME/cache/spillover instead of truncating when no sandbox env is active
Sessions that never ran a terminal command (MCP-only, cron, gateway)
have no active sandbox environment, so maybe_persist_tool_result()
got env=None and fell through to the inline-truncate fallback --
a 467K MCP result was cut to a ~1.3K preview with no file written
('Full output could not be saved to sandbox').

Now the host-side cases (env=None or the local backend) write the
spill file directly to $HERMES_HOME/cache/spillover/<id>.txt,
alongside the other Hermes-owned caches instead of littering /tmp.
Remote backends (docker/ssh/modal/daytona) keep the in-sandbox
env.execute() write since read_file resolves in-sandbox there.

Cleanup: the gateway housekeeping loop prunes spillover hourly with
the other media caches, and a once-per-process best-effort prune on
first spill covers CLI-only installs.
2026-08-18 02:12:13 -07:00
Victor Kyriazakos 22f0f22298 fix(cron): manual runs no longer silently drop media attachments
Field report (enterprise, v0.20.0): cron jobs delivering text + PDF/image
attachments to Slack DMs deliver both on scheduled ticks but text-only on
manual `hermes cron run <job-id>`. Same box, same token, same scopes —
the divergence is process context and error visibility, not credentials.

Three defects, one bug class (attachment failures invisible + policy
divergence between the gateway process and standalone processes):

1. Standalone lane swallowed warnings: platform standalone senders
   (Slack files_upload_v2, Discord, ...) report per-file upload failures
   in result['warnings'] while returning success=True for the delivered
   text leg. _deliver_result only read result['error'], so the run was
   marked ok and the attachment vanished without a trace. Warnings now
   surface into delivery_errors (and the job's last_error).

2. Live-adapter lane swallowed media failures: _send_media_via_adapter
   logged failures at WARNING and returned None. It now returns per-file
   error strings and _deliver_result records them — text-delivered-but-
   attachment-failed is a visible partial failure on BOTH lanes.

3. Media-policy env bridge was gateway-only: gateway.strict /
   media_delivery_allow_dirs / trust_recent_files were translated from
   config.yaml to the env vars validate_media_delivery_path reads ONLY in
   gateway startup. A CLI-process manual run filtered attachment paths
   under a different policy — in strict/allowlisted deployments the exact
   reported symptom (scheduled delivers, manual drops, silently). The
   translation now lives in gateway/media_policy.apply_media_policy_env
   (idempotent, env-wins, never raises); gateway startup delegates to it
   and _deliver_result applies it before filtering. Attachments dropped
   by the policy filter are also reported in the run status instead of
   only a stderr WARNING.

On v0.20.0 specifically the failure was double-blind: the pre-9cf2cbd382
isinstance(resp, dict) gates meant upload failures were undetectable in
the sender AND unsurfaced by the scheduler. 9cf2cbd382 (in 2026.8.13)
fixed detection; this fixes visibility and policy parity.

8 new tests (tests/cron/test_media_delivery_parity.py): warnings→errors,
clean-delivery control, media-reaches-sender control, live-adapter
failure/dropped-path reporting, bridge helper semantics, strict+allowlist
end-to-end in a non-gateway process, and the .env-strict/config-allowlist
split that reproduces the field symptom. Mutation check: disabling the
warnings loop and the bridge fails exactly the 2 guarding tests.
2026-08-17 17:19:50 -07:00
k0rnacki 1352439036 fix(api_server): recovery net must never recover the advertised virtual alias
The #35314 empty-dispatch recovery cache accepts any previously
resolved model string, including the advertised virtual model
(hermes-agent by default) — which is never a dispatchable model.
Guard both directions: never store the alias in _last_resolved_model,
and never serve a cached value equal to it. The alias now survives at
most one turn and cannot propagate across turns through the cache.

Class bug behind #79101; complements the session-row symptom fixes
(#72739, #79102).
2026-08-17 16:54:01 -07:00
Andy Lo-A-Foe 152b1940d7 fix(api-server): stop persisting the virtual model alias as a session's model
Salvaged from #72739 (net diff onto current main; the read-side legacy-row
guard now flows through the shared _stored_session_model() helper that
landed in #88751, keeping one resolver for both chat sites).

POST /api/sessions persisted the advertised virtual alias (hermes-agent)
whenever the request omitted model or echoed the alias back; later turns
replayed it upstream as a real model id and every turn on the session
failed. Null the alias in _session_runtime_request_from_body() — shared by
session create, chat, chat-stream, and model-lock — and stop the
create-handler's raw-body fallback from bypassing that normalization.
2026-08-17 16:54:01 -07:00
Teknium 189de16280 fix(peer): live-verified cross-gateway bot DMs — session shape + virtual-model replay
Two bugs found by a REAL two-gateway live test (two isolated HERMES_HOMEs,
bravo running the api_server platform, alpha's agent autonomously running
`hermes peer dm` from its Bot Chat protocol; reply relayed correctly and
persisted in bravo's canonical Bot Chat):

1. peer dm parsed the session-create response flat, but api_server wraps
   the row: {"object": "hermes.session", "session": {...}} — every first DM
   to a fresh peer failed with "Peer did not return a session id" (and the
   orphaned Bot Chat then 400'd retries with duplicate-title). Parse the
   wrapped shape; the test fake now mirrors the real response shape so this
   class can't pass green again.

2. api_server: a session created with no model persists the advertised
   virtual model ("hermes-agent") on the row; session chat then replayed it
   as a REAL model id and the provider 400'd ("hermes-agent is not a valid
   model ID"). _request_agent_overrides already filters the virtual model
   for per-request bodies — apply the same filter to the stored session
   model at both chat sites (sync + stream), so it means "gateway default"
   exactly like the request-body path.

Live E2E transcript (bravo's Bot Chat, via /api/sessions/{id}/messages):
  user:      Message from 🤖 alpha (@alpha): What is your callsign?
  assistant: CALLSIGN-BRAVO-7
peer cmd unit suite 10/10 with the corrected fake.
2026-08-17 16:34:22 -07:00
Teknium 6f2dbf3a05 fix(gateway): widen scope-aware session DB resolution to the runner and release cached handles
Follow-up to the salvaged #88632 (Jack Lau) fix for #88532, extending the
same repair to the sibling frozen-at-init handle and closing the handle
lifecycle gap the per-path cache introduces.

1. GatewayRunner._session_db had the identical bug class: bound once as
   AsyncSessionDB(SessionDB()) in __init__ on the root home, while /resume,
   /title, /history and session search all execute inside
   _profile_runtime_scope on a multiplexed gateway. Convert it to the same
   property-with-pin pattern: per-access resolution of _default_db_path(),
   one cached AsyncSessionDB per resolved path under a lock, and assignment
   preserved as an explicit pin (many suites install fakes or None).
   Construction-time priming keeps the #88235 init-failure broadcast at
   startup.

2. Handle lifecycle: the per-path caches accumulate one open SessionDB per
   profile served, but the shutdown path closed only store._db /
   runner._session_db - which now resolve just the shutdown task's own
   (root) scope. Secondary profiles' handles would strand their WAL write
   locks until process exit, recreating the abandoned-handle leak
   b454e4da76 fixed and breaking --replace restarts with 'database is
   locked'. Add close_all_db_handles() / close_all_session_db_handles()
   sweeps and call both from the gateway teardown path. SessionDB.close()
   is idempotent, so the root handle being closed by both the legacy loop
   and the sweep is safe.

Tests: sweep coverage plus a runner-property scope/pin/cache test in
tests/gateway/test_multiplex_session_db_profile_scope.py.
2026-08-17 16:34:10 -07:00
Jack Lau 17ba992108 fix(gateway): resolve the session DB inside the active profile scope
Fixes #88532.

A multiplexed gateway serves every profile from one process, but
SessionStore bound a single SessionDB during __init__:

    self._db = SessionDB()

SessionDB(db_path=None) resolves _default_db_path() at call time and
does follow the context-local HERMES_HOME override, so the path
machinery was already correct. The problem was when it ran: at
construction, on the process's own root home, long before any inbound
event enters _profile_runtime_scope. Every profile's rows therefore
landed in the root state.db, even though the scope had redirected
get_hermes_home() correctly for the turn (that helper's own docstring
lists "sessions" among what it scopes).

The rows still carry the right profile_name, stamped from
source.profile by the same handler, so nothing in the data looks wrong.
The only visible symptom is the desktop listing a profile's session
under the default bot: _open_session_db_for_profile opens
profiles/<name>/state.db, which never received the write.

Look the handle up through a property instead, resolving the active
scope per access and caching one handle per resolved path so a hot
inbound path opens SQLite once per profile rather than once per
message. Construction stays under the cache lock so a concurrent first
message on a profile cannot open and then leak a second handle.

Assignment is preserved as an explicit pin, which is what the existing
suites rely on when they install a fake handle or disable the DB with
store._db = None, and a pin keeps winning across scope changes.

Behavior is unchanged when no profile scope is active, so single-profile
gateways resolve exactly the path they did before. This does not migrate
rows that already landed in the root store; those stay where they are.
2026-08-17 16:34:10 -07:00
Gille 68a713385e fix(gateway): preserve configured context window 2026-08-17 15:46:23 -07:00
Teknium 481156139d feat: misfire catch-up for external cron providers
When an external scheduler (Chronos on hosted deployments) cannot
deliver a fire — dead loopback hop at fire time, retry budget exhausted
— the job's next_run_at stays parked in the past and nothing ever runs
it: external providers have no local tick loop, so the day is silently
lost even if the gateway heals minutes later (4 consecutive nightly
misses in the field).

fire_overdue_jobs() in cron/scheduler_provider.py, called from the
gateway housekeeping loop every 5 minutes:

- No-op for the built-in ticker (its tick loop already self-heals
  past-due jobs) and when cron.misfire_grace_minutes <= 0.
- Waits out a grace window (default 10 min) so the external scheduler's
  own retry backoff gets first right to deliver.
- Claims via the provider's claim_fire (store CAS — a concurrent late
  external retry is de-duplicated) and runs fire_claimed in a daemon
  thread, mirroring the webhook admission pattern, so housekeeping
  never blocks for the length of an agent run. Provider re-arm logic
  (Chronos NAS one-shots) runs exactly as for a normal fire.

Docs: cron.md section + cron.misfire_grace_minutes reference.
2026-08-17 11:42:25 -07:00
Teknium cb1b1da219 fix: surface missed cron fires as last_fire_error on the job record
On hosted deployments a scheduled fire that cannot be forwarded to the
gateway api_server (dead 8642 listener, gateway down) was invisible
outside gui.log: no execution row is created because the claim never
happens, so `cronjob list` showed a healthy job that silently missed
days of scheduled runs (4 consecutive nightly misses in the field,
diagnosed only by log grep).

Changes:
- cron/jobs.py: note_fire_forward_failure() durably stamps
  last_fire_error ({at, detail}) on the job record; mark_job_run clears
  it on the next successful run so it always describes current
  auto-fire health (mirrors preflight_alerted/drift_alerted).
- hermes_cli/web_routers/cron.py: the dashboard fire webhook stamps the
  job on the gateway-unreachable path, best-effort (never disturbs the
  503/Retry-After retry contract or the OOF-266 intentional-stop drop).
- tools/cronjob_tools.py: _format_job carries last_fire_error so the
  agent-facing cronjob list surfaces it.
- hermes_cli/cron.py: `hermes cron list` prints a red
  "Missed scheduled fire" line.
- web/: dashboard CronPage renders the miss; api.ts type updated.
- gateway/run.py: one-time startup warning when an external cron
  provider is active but the api_server adapter is not running (the
  fire path is dead-on-arrival; most common cause is API_SERVER_KEY
  missing from an unsupervised gateway relaunch).
- website/docs: cron doc section on missed fires.
2026-08-17 11:29:10 -07:00
kshitij 979ca57a50 Merge pull request #88244 from kshitijk4poor/fix/handoff-cleanup-race
fix: prevent handoff leg data loss + surface state.db corruption to users
2026-08-17 17:41:44 +05:30
Shannon Sands 4beca7a943 feat(kanban): memory-aware dispatch guard + memory-derived default concurrency cap (OOF-30)
Two production incidents (OOF-77 "larrikin-lollies", OOF-30
"synclare-task-manager") followed the same shape: no
kanban.max_in_progress configured, a busy board, and a 1 GiB hosted VM.
The dispatcher fanned out 26-31 concurrent workers, the host went into
swap-thrash/OOM, and the whole machine — dashboard included — became
unreachable. NAS restart loops then masked the problem: each restart
"recovered" briefly before the kanban dispatcher immediately respawned
unbounded workers.

Building on the cherry-picked max_in_progress-across-both-lanes fix
(PR #28695, credit @Dusk1e), this adds two complementary safeguards to
hermes_cli/kanban_db.py:

1. Memory-DERIVED default concurrency cap. When kanban.max_in_progress
   is unset, resolve_max_in_progress() derives a default of
   clamp(MemTotal / 512 MiB, 2, 8) — e.g. 2 workers on a 1 GiB VM,
   8 on 4 GiB+. Explicit config always wins in either direction. On
   hosts where total memory can't be read (macOS/Windows dev machines),
   the default stays None (no cap — unchanged behaviour). Wired into
   both dispatch entry points (gateway/kanban_watchers.py and
   hermes kanban dispatch) so behaviour matches regardless of path.

2. Live memory-PRESSURE guard inside dispatch_once. A static cap can't
   see the host's actual memory state (other tenants, bloated
   long-lived workers). The dispatcher now samples system memory each
   tick via gateway.lifecycle_ledger.sample_memory() and classifies it
   with gateway.memory_status.classify_pressure() (same thresholds as
   the dashboard memory banner and OOM-suspicion heuristics from
   NS-608/NS-656): critical -> spawn nothing this tick; elevated ->
   at most one new worker; unknown -> no restriction (fail-open).
   Reclaim/promotion bookkeeping still runs under pressure, and
   deferred tasks stay queued — nothing is dropped. Restriction is
   surfaced on DispatchResult.memory_pressure and logged.

Tests: tests/hermes_cli/test_kanban_memory_guard.py (14 tests) covers
the derived cap (floor/ceiling/fail-open/explicit-config-wins), the
pressure classifier, and dispatch behaviour under critical/elevated/
unknown pressure including defer-not-drop and bookkeeping-still-runs.
An autouse fixture in tests/conftest.py pins the memory sample to
"no data" suite-wide so existing dispatch tests don't depend on the
CI runner's live memory state (opt-out marker: real_memory_guard).
2026-08-17 14:16:21 +05:30
kshitij 59f302fef9 fix: prevent handoff leg data loss + surface state.db corruption to users
Two data-loss bugs reported by users:

1. /handoff CLI→gateway race (#88234): After /handoff completed, CLI
   cleanup called finalize_session on the session the gateway just
   reopened. This set end_reason on a row the gateway was actively
   writing to, causing the handoff leg to vanish from session history
   and breaking session_search recall. Fix: add _handed_off_session_ids
   module-level set (mirrors _single_query_finalize_attempted_session_ids
   pattern). _handle_handoff_command registers the session_id on
   completion; _should_emit_cleanup_session_finalize and
   _emit_interrupted_session_end check it before firing.

2. state.db corruption silent failure (#88235): When SessionDB init
   failed at gateway startup, the error stayed in logs — messages
   flowed but nothing was persisted, with no user-visible indication.
   Fix: store _session_db_init_error on GatewayRunner, broadcast a
   recovery-guidance message to all home channels via
   _send_session_db_warning_notifications() after the gateway connects.
   Also improved the 'corrupt' persistence cause wording in
   _format_turn_completion_explanation to include the full recovery
   path (hermes doctor --fix, sqlite3 .recover, backups).

Tests: 6 new tests for handoff cleanup race, 3 for corruption wording.
All existing CLI/turn-completion tests pass.
2026-08-17 13:19:58 +05:30
Teknium bd4b709258 Inspired by Copilot CLI: /rollback keeps user hand-edits by default
Copilot CLI 1.0.78 reworked /rewind to restore only the files the agent
changed, 'skipping any file whose contents no longer match what Copilot
last wrote'. This ports that protection to Hermes checkpoints:

- tools/checkpoint_manager.py: per-project agent-write ledger
  (sha256 of every landed write_file/patch), safe_restore_plan()
  classifier, and restore(safe=True) that reverts only agent-authored
  changes, deletes agent-created files, and preserves user hand-edits.
  Empty ledger (pre-existing stores) falls back to the classic full
  restore.
- run_agent.py: feed the ledger from _record_file_mutation_result on
  every landed mutation (zero new hooks; rides the existing verifier).
- CLI + gateway /rollback: safe mode is the default; --all/--force
  restores everything; skipped files are reported with a hint.
- 17 locales: new gateway.rollback.kept_user_edits key.
- Docs: checkpoints-and-rollback.md updated.
- Tests: 7 new cases incl. user-edit preservation, post-agent user
  tweaks, agent-created file removal, empty-ledger fallback.
2026-08-16 22:08:47 -07:00
Shannon Sands d66341ab28 fix(status): strict writer-identity ownership for aggregated platform entries (OOF-3)
The freshness window (updated_at >= live process create_time - 2s) had a
P1 boundary hole: a stale failure written by the PREVIOUS process
immediately before a fast restart landed inside the slack and was
aggregated; if that platform was then removed, the new process never
replaces the entry and NAS stays degraded indefinitely.

Replace clock heuristics with persisted writer identity:

- write_runtime_status now stamps every platform entry with the writing
  process's (writer_pid, writer_start_time) — the same PID-reuse
  fingerprint the liveness checks use, so a recycled PID never
  masquerades as the original writer.
- The aggregation ownership filter requires exact equality between an
  entry's stamp and the profile's validated live gateway process
  (get_runtime_status_running_pid + _get_process_start_time). No slack,
  no timestamps. Legacy entries without a stamp fail closed.
- Writer stamps are process recon (same class as the auth-gated
  gateway_pid) and are stripped from all /api/status projections, both
  active-profile and merged cross-profile entries.

Near-boundary regression test: prior-process entry stamped 100ms before
restart is excluded; recycled-pid-different-fingerprint excluded;
legacy no-stamp excluded; current-process entry kept.
2026-08-16 20:32:09 -07:00
Shannon Sands f755ed5e90 fix(gateway): surface multiplex profile failures (OOF-3) 2026-08-16 20:32:09 -07:00
Shannon Sands c127d0c3b1 fix(gateway): attribute scoped credential lock conflicts to the owning profile (OOF-3)
Scoped credential locks (Telegram bot token, Discord bot token, etc.) are
machine-global, but the conflict error only reported the holder's PID:

    Telegram bot token already in use (PID 559). Stop the other gateway first.

On multi-profile hosts (e.g. hosted instances running 13 profiles), a bare
PID gives the operator no way to tell WHICH profile owns the credential —
the exact failure mode observed on zerocool-9781, where the 'default'
profile was misconfigured with the same bot token as 'lead-gen-outreach'
and logged an unattributable conflict every ~5 minutes (4,602 rows).

Fix:
- acquire_scoped_lock() now stamps a 'profile' label on lock records,
  inferred from the process HERMES_HOME (<root>/profiles/<name> layouts,
  'default' for the root home). Omitted when not inferable.
- New scoped_lock_owner_label() resolves the owning profile from a lock
  record: prefers the explicit field, falls back to inferring from the
  persisted hermes_home for locks written before the field existed.
  Labels are validated against the profile-id grammar before use (lock
  files are plain JSON on disk and the label flows into log lines and a
  suggested CLI command).
- _acquire_platform_lock() conflict message now names the owning profile
  and gives the correct remedy:

    Telegram bot token already in use by the 'lead-gen-outreach' profile
    gateway (PID 559). Stop that gateway first
    (hermes --profile lead-gen-outreach gateway stop).

  Records with no attribution signal keep the original PID-only wording.

Testing:
- New TestScopedLockOwnerLabel suite covering label inference (named,
  Docker, root/default, unknown layouts), grammar validation, explicit-
  field preference, hermes_home fallback, and legacy/malformed records.
- acquire_scoped_lock tests for profile stamping and omission.
- Adapter-level tests for profile-attributed, legacy-home-inferred, and
  PID-only conflict messages.
- 76/76 targeted gateway tests pass; broad gateway suite failures are
  baseline-identical (verified via git stash comparison). Ruff clean.
2026-08-16 20:32:09 -07:00
liuhao1024 8dc427949f fix(gateway): accept CJK full-width punctuation as MEDIA path terminators
MEDIA_TAG_CLEANUP_RE (and MEDIA_EXTENSIONLESS_TAG_RE) only recognized
ASCII terminators after a MEDIA:<path> tag. Chinese-language agent
output naturally writes MEDIA:D:\...\zhibao.pdf(782.6 KB)or ...pdf:内容 —
the full-width punctuation failed the trailing lookahead and the
attachment was silently dropped (cron even reported 'delivered') (#88038).

Both lookaheads now accept a CJK full-width terminator set (()〈〉《》:,。;
!?、curly quotes【】) alongside the ASCII set. The #68773 adjacent-tag
splitting guard is covered by a regression test.
2026-08-16 19:54:42 -07:00
Adolanium fccf2b718e fix(gateway): complete /loop ticks after streamed already_sent turns
Streamed replies return None so the adapter does not send twice.
The /loop hook then saw empty text and never ran, so
awaiting_response stayed true and later ticks never fired.

Stash the delivered text on the event and use it for the post-turn
hooks. /goal uses the same path.

Tests: tests/gateway/test_loop_command.py
2026-08-16 06:31:07 -07:00
Merge_Conflict - Pasi 309cf2c5e2 fix: honor JSON-array string forms for skills.disabled and agent.disabled_toolsets
`hermes config set` and JSON-mode editor saves store lists as quoted
strings (e.g. '["skill-a","skill-b"]' or "['memory']"). Both disable
filters treated such a string as a single name, so curated disable
lists silently filtered nothing with zero diagnostics.

Add parse_config_string_list() in agent.skill_utils and use it in
_normalize_string_set (skills.disabled / platform_disabled) and at
every agent.disabled_toolsets read site: tools_config resolve +
reconcile, CLI, gateway agent construction (both sites), cron
scheduler, and prompt_size. A scalar string still names a single
entry (#13026); malformed JSON falls back to the single-name
behavior instead of raising.

Fixes #86661
2026-08-16 06:24:56 -07:00
poisdahl 7ca1987459 Merge upstream main into PR 81234 2026-08-16 12:20:50 +02:00
fangliquanflq 095d25c612 fix(gateway): keep persisted model routes consistent 2026-08-16 02:03:31 -07:00
fangliquanflq b3df990832 fix(gateway): offload hygiene streak persistence 2026-08-16 02:02:06 -07:00
fangliquanflq 39d2d858fd fix(gateway): persist hygiene failure cooldown rung 2026-08-16 02:02:06 -07:00
webtecnica b7936c892d fix(gateway): catch BaseException in _process_message_background to notify on SystemExit
The fire-and-forget handler only caught asyncio.CancelledError and
Exception, so a SystemExit/KeyboardInterrupt escaping a turn (e.g. a
plugin calling sys.exit() in a tool call or summary-LLM path) skipped
the user-facing failure notification and surfaced only as 'Task
exception was never retrieved' — radio silence for the user.

Catch BaseException instead; send the failure notification first, then
re-raise SystemExit/KeyboardInterrupt to preserve shutdown semantics
for the loop's own signal handling. Other BaseExceptions stay contained.

Closes #86651
2026-08-16 01:51:10 -07:00
Christopher 3a61fa86fa fix(gateway): keep broad connection phrases out of the provider-error gate
Fixes #86570
2026-08-16 01:50:44 -07:00
Christopher 5cfa2b2f0f fix(gateway): surface actionable message for local model server connection errors 2026-08-16 01:50:44 -07:00
fangliquanflq faaf2ae093 fix(gateway): read routed profile model config 2026-08-16 01:49:53 -07:00
liuhao1024 f13f3401a1 fix(webhook): authenticate Linear deliveries via linear-signature HMAC (#87348) 2026-08-16 01:45:37 -07:00
fangliquanflq 9219cd3944 fix(gateway): isolate post-turn loop failures 2026-08-16 01:39:14 -07:00
fangliquanflq 2f6bbfbcbc fix(gateway): release loop ticks after empty responses 2026-08-16 01:39:14 -07:00
Teknium 763b10c320 fix(gateway): run session-finalize plugin hooks off-loop and bounded
Session-finalize hooks ran synchronously on the gateway event loop from
three call sites (shutdown drain, session-expiry watcher, /new reset).
A plugin hook doing heavy blocking work froze the whole loop: adapter
heartbeats stopped, the drain machinery could not run, and systemd
eventually SIGKILLed the process mid-export. Observed live on a
multi-day 4.7G session where the nemo_relay observability plugin
serialized a full-session ATIF trace inside on_session_finalize.

Changes:
- gateway/run.py: new GatewayRunner._finalize_session_off_loop()
  dispatches hermes_cli.lifecycle.finalize_session via the gateway
  executor under asyncio.wait_for (10s budget), mirroring
  _cleanup_agent_resources_off_loop (#53175). Shutdown finalize and
  the session-expiry watcher now use it.
- gateway/slash_commands.py: /new reset path uses the same helper.
- plugins/observability/nemo_relay: ATIF export is now bounded
  (HERMES_NEMO_RELAY_ATIF_EXPORT_TIMEOUT_S, default 30s) and skipped
  entirely for sessions whose Relay scope operations already errored
  (their exporter state is unreliable and the export can be
  pathologically slow).
- tests/gateway/test_finalize_session_off_loop.py: regression tests
  proving the loop stays live under a wedged hook and the budget is
  enforced.
2026-08-15 14:20:27 -07:00
poisdahl 67ee481621 fix(gateway): preserve pending transcript state on failed rewinds 2026-08-15 22:43:00 +02:00
poisdahl 3f075d41dd Merge remote-tracking branch 'origin/main' into codex/81234-live-main-final
# Conflicts:
#	tui_gateway/methods_prompt.py
#	tui_gateway/server.py
2026-08-15 12:05:14 +02:00
Teknium ca5c04818b feat: bare /save shows a usage card instead of exporting
Per review: /save with no arguments prints usage (formats, filename,
redact, examples) and writes nothing; the export requires an explicit
format. Unknown formats print the error plus the same usage card. One
shared SAVE_USAGE string in hermes_cli/session_export.py serves both the
CLI and gateway handlers. args_hint updated to <json|md|html> to reflect
the now-required format. Tests: bare-save and bad-format usage cases
added; location tests pass an explicit format.
2026-08-15 02:04:49 -07:00
Teknium e5a2c80c0e fix: harden /save for test doubles, async session-store boundary, snapshot shape pin
Three CI-red follow-ups on the /save rework:

- cli.py save_conversation: getattr-guard _session_db/session_id so
  SimpleNamespace/object.__new__ test doubles (and any embedder passing a
  minimal stub) don't AttributeError (pitfall #17 pattern).
- gateway/slash_commands.py: route through the awaited
  async_session_store.get_or_create_session boundary — the architecture
  test forbids raw session_store calls in async gateway code.
- tests/cli/test_save_conversation_location.py: /save now emits the
  canonical export_session payload shape; the session id key is "id"
  (was "session_id" in the legacy snapshot format) — update the pin.

The slice-8 lost-and-found failure was pre-existing on main and is fixed
there by f2a30fa400 (test: derive lost-and-found synthetic width from the
live schema); picked up via rebase.
2026-08-15 02:04:49 -07:00
Teknium 26aa12337a feat: /save exports the current session as json, md, or html on all platforms
Rework of salvaged PR #6372 (@ag9920) onto current main:

- /save promoted from CLI-only JSON snapshot to a cross-platform session
  export: `/save [json|md|html] [filename] [redact]` on CLI and every
  gateway platform (sent as a document via adapter.send_document).
- Rendering routes through the existing shared renderers
  (hermes_cli/session_export.py + session_export_html.py) instead of the
  PR's new hermes_state formatter — new helpers normalize_save_format /
  render_session_for_save / default_save_filename are shared by both
  surfaces.
- `redact` arg runs the export through the force-mode secret redaction
  pass (session_export_md.redact_session_data) before writing.
- Gateway handler awaits AsyncSessionDB correctly, sanitizes user-supplied
  filenames with basename, and lands in gateway/slash_commands.py (the
  handlers moved out of gateway/run.py since the PR was authored).
- /export stays profile export (name collision resolved: session export
  lives on /save).
- Slack 50-slash cap curation: /platform moves to the /hermes-only set to
  free a native slot for /save (parity test updated rationale comment).
- Folds in PR #62268 (@briandevans): None title/model coalescing in the
  single-session HTML export.

Closes #4249. Closes #51200.
2026-08-15 02:04:49 -07:00
ag9920 23ad35b7fa feat: Add /export command for session export (Markdown/JSON) to CLI and Gateway 2026-08-15 02:04:49 -07:00
Nicolas Formenton c75d835555 feat(desktop): mark a session as unread/read with a persisted watermark 2026-08-15 01:21:40 -07:00
doncazper 6a375a24a5 fix(gateway): timestamp shared stderr output 2026-08-15 01:04:19 -07:00
Teknium fbaea9bddc feat(sessions): generic 'hidden' session flag (sidebar-hide, still resumable) (#86797)
* feat(sessions): generic 'hidden' session flag (sidebar-hide, still resumable)

Adds a source-orthogonal, archive-orthogonal 'hidden' session flag meaning
'don't show in the global Sessions sidebar, but stay fully resumable by the
surface that owns it'. Mirrors the existing archived/pinned capability end to
end, so it's a generic widening (any plugin that owns its own session lifecycle
- kanban, Bot Mode, future plugins - can keep its sessions out of the shared
recents list) rather than a per-plugin special-case.

- Schema: hidden INTEGER NOT NULL DEFAULT 0 on sessions (additive; lands on
  existing DBs via the declarative _reconcile_columns ADD COLUMN path, same as
  archived/pinned - no version-gated migration).
- DB: SessionDB.set_session_hidden(session_id, hidden) (clones set_session_pinned
  incl. the compression-lineage recursive CTE); list_sessions_rich gains
  include_hidden=False, appending 's.hidden = 0' by default so hidden rows drop
  from every listing path (and the REST sidebar endpoints inherit it with no
  change).
- Gateway: session.set_hidden RPC (mirrors session.title); session.create accepts
  hidden=true, deferred via pending_hidden and applied in _ensure_session_db_row
  when the row is lazily created (mirrors pending_title).
- REST parity: PATCH /api/sessions/{id} accepts+bool-validates 'hidden' ->
  set_session_hidden; _session_response exposes it.

Enables Hermes-Bot-Mode to hide canonical 'Bot Chat' sessions from the sidebar
(NousResearch/Hermes-Bot-Mode#46) WITHOUT retagging source (which would mis-set
the agent platform). Bot Chats keep source=desktop. Gateway RPC needs a
SERVE-backend restart to take effect live. 1 focused test (default-exclude /
include_hidden / unhide round-trip).

* fix: teach lost-and-found recovery about the 55-column sessions layout

Adding the 'hidden' column makes the current sessions table 55 columns. The
SQLite lost-and-found recovery classifier keys off the physical field count
(SESSIONS_LAYOUT_NFIELDS) to identify a salvaged sessions row, so a recovered
current-layout row (nfield=55) would otherwise be unrecognized and dropped.
Add 55 to the frozenset (54/52 stay as historical prefixes) and update the
column-count assertions + synthetic current-layout insert in the recovery test.

---------

Co-authored-by: Teknium <teknium1@users.noreply.github.com>
2026-08-15 00:31:37 -07:00
Teknium 059e346422 fix(gateway): abort-aware parallel startup connects (restart race parity) 2026-08-14 21:57:41 -07:00
Teknium c22815fcaa fix(gateway): cap Telegram cold-start connect so startup reaches running fast (#85993)
CI / Docs Site (push) Has been cancelled
CI / Detect affected areas (push) Has been cancelled
CI / Python tests (push) Has been cancelled
CI / OS-specific tests (push) Has been cancelled
CI / Python lints (push) Has been cancelled
CI / JS & TS checks (push) Has been cancelled
CI / Installer tests (push) Has been cancelled
CI / Desktop E2E (push) Has been cancelled
CI / Deny unrelated histories (push) Has been cancelled
CI / Check contributors (push) Has been cancelled
CI / Check uv.lock (push) Has been cancelled
CI / Check no committed infographics (push) Has been cancelled
CI / package-lock.json diff (push) Has been cancelled
CI / Lint Docker scripts (push) Has been cancelled
CI / Supply-chain scan (push) Has been cancelled
CI / Review label gate (push) Has been cancelled
CI / OSV scan (push) Has been cancelled
CI / All required checks pass (push) Has been cancelled
CI / CI timing report (push) Has been cancelled
Deploy Site / deploy-vercel (push) Has been cancelled
Deploy Site / deploy-docs (push) Has been cancelled
Docker Build, Test, and Publish / Detect affected areas (push) Has been cancelled
Docker Build, Test, and Publish / build (amd64, type=gha,scope=docker-amd64, type=gha,mode=max,scope=docker-amd64, linux/amd64, ubuntu-latest) (push) Has been cancelled
Docker Build, Test, and Publish / build (arm64, type=gha,scope=docker-arm64, type=gha,mode=max,scope=docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Docker Build, Test, and Publish / publish (amd64, type=gha,scope=docker-amd64, type=gha,mode=max,scope=docker-amd64, linux/amd64, ubuntu-latest) (push) Has been cancelled
Docker Build, Test, and Publish / publish (arm64, type=gha,scope=docker-arm64, type=gha,mode=max,scope=docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Docker Build, Test, and Publish / merge (push) Has been cancelled
auto-fix lint issues & formatting / Generate eslint --fix patch (push) Has been cancelled
auto-fix lint issues & formatting / Apply patch (push) Has been cancelled
The initial (pre-running) connect awaited during gateway startup now uses
a capped 45s budget for Telegram instead of the full 180s (#67498) budget.
On timeout the platform is queued for the reconnect watcher, which retries
with the full budget and is_reconnect=True (preserving the offline update
queue, #46621). Combined with the parallel startup connects, an unreachable
Telegram no longer holds the whole gateway out of the running state.
2026-08-14 21:57:41 -07:00
EvanProgramming d86c67dc7e fix(gateway): connect messaging platforms in parallel at startup (#83791)
GatewayRunner.start() previously awaited each platform's connect() (with its
own timeout) in a serial for-loop. A single slow/failing platform (e.g.
Telegram behind a dead proxy) delayed every later platform's connect by a full
timeout window, cascading one platform's failure onto WeChat/QQ/etc.

Now the slow connect() calls run concurrently via asyncio.gather while the
serial pre-filter (checks, adapter creation, handler wiring) and the
single-threaded result aggregation (shared-state mutation, error handling)
are unchanged. A failing platform no longer blocks the others.

Adds regression tests proving connect() calls overlap and that one failing
platform leaves the others connected.
2026-08-14 21:57:41 -07:00
Teknium 366fd70f98 fix(gateway): registered_names() honors profile scope like is_registered()
The salvaged registered_names() from PR #71582 predates the scoped
platform registry: it read only the process-global _entries/_deferred
maps, but plugin platforms register their deferred loaders under a
profile scope. Result: `hermes tools enable a2a --platform a2a` still
rejected the platform. Union the current-scope maps with the global
ones, mirroring is_registered()'s semantics, under the registry lock.
2026-08-14 21:56:33 -07:00
tachyon-r 7a6b8917f7 fix(tools): recognize discovered plugin platforms 2026-08-14 21:56:33 -07:00
kshitij 24573b396b refactor: bind gateway anti-growth guard to locals, trim overlong comment
Simplify-code pass: gateway/run.py called estimate_messages_tokens_rough
6x on the same data in the anti-growth guard (condition + warning f-string).
Bind to _hyg_in_toks/_hyg_out_toks locals like the conversation_compression.py
guard already does. Also trim the comment from 10 lines to 4 (keep the WHY,
drop the WHAT) and remove an extra blank line before TestCompactedTurnsStaySearchable.
2026-08-15 10:22:47 +05:30