1692 Commits

Author	SHA1	Message	Date
fangliquan	91fb175188	fix(gateway): preserve systemd handoff recovery	2026-08-22 15:27:26 +05:30
fangliquanflq	5b024c7ccc	fix(gateway): make systemd the sole restart owner	2026-08-22 15:27:26 +05:30
fangliquan	bf3a0bb99d	fix(gateway): isolate supervised watcher contexts	2026-08-22 15:25:50 +05:30
Jack Lau	e173720774	fix(gateway): give supervision exhaustion an owner for queued platforms ... Review of #90448 by @andrexibiza: adding _ensure_reconnect_watcher_running() to the already-queued branch of a fatal callback is still an event-coupled check. It needs a later fatal error from some other platform to arrive, and #81036 makes that less likely rather than more -- it publishes the queue before disconnect and drops the failed adapter from the live map, so after the watcher's supervised restart budget is spent there may be no adapter left to emit the event recovery is waiting on. That is the state #72366 (salvage of #71867 by @ygd58) restored supervision to close: queued work exists, the watcher is dead, and nobody owns the invariant. Supervision being finite is correct; having no owner past the budget is not. _spawn_supervised now takes on_give_up, invoked when it abandons a task -- the supervisor is the only thing that knows it has. The reconnect watcher uses it to hold: while _running and _failed_platforms is non-empty, either a reconnect watcher is live or a bounded respawn is scheduled. Empty queue: leave it down and log; the enqueue path spawns a fresh watcher the moment something depends on one. Non-empty: a bounded slow tier at _RECONNECT_WATCHER_SLOW_RETRY_SECS (300s) for _MAX_SLOW_WATCHER_RESPAWNS (6) attempts, standing down early if the queue drains or a watcher returns on its own. Exhausted: one loud error naming the platforms left unattended. The ceiling is (1 + _MAX_SUPERVISED_RESTARTS) x (1 + _MAX_SLOW_WATCHER_RESPAWNS) spawns -- 42 across at least half an hour -- because each slow attempt hands the watcher a fresh supervised budget. A test asserts that ceiling so it cannot quietly become a restart loop. Deliberately NOT included: requesting a process restart when the slow tier is also exhausted. Taking down every healthy platform to heal a sick one is a blast-radius policy decision for a maintainer. Two things this turned up: - _spawn_supervised did not thread on_give_up through its own backoff respawn, so the callback was lost after the first restart and the give-up branch had no owner at exactly the moment it needed one -- the same defect the on_spawn docstring warns about, one parameter over. - Three call sites repeated the (factory, name, on_spawn) triple, whose on_spawn half is load-bearing. They now go through _spawn_reconnect_watcher(). _supervised_backoff() names the previously-inline exponential schedule so the exhaustion tests can collapse it; production behaviour is unchanged. Refs #90386	2026-08-22 15:25:45 +05:30
Jack Lau	92018e76a8	fix(gateway): heal a dead reconnect watcher when the platform is already queued ... _ensure_reconnect_watcher_running() exists for one situation: the reconnect watcher has exhausted _MAX_SUPERVISED_RESTARTS, so _spawn_supervised has logged "giving up restarts" and will never bring it back on its own (#70344, and the supervised-restart half of #71758). It had exactly one call site, inside the newly-queued branch of _queue_retryable_fatal_platform. That branch is unreachable for a platform already in _failed_platforms, which is the only kind of platform the watcher can have been retrying long enough to burn five rapid restarts on. So the backstop could not fire in the one state it was written for. The failure is silent by construction. The early return logs nothing, so there is no "queued for background reconnection" line. The stranded check in _handle_adapter_fatal_error_detached deliberately treats a queued platform as safe, so the gateway does not exit for the service manager either. With another platform still connected, self.adapters is non-empty and the "gateway staying alive, watcher will retry in background" branch is skipped too. A retryable fatal error can therefore produce a single ERROR line and then nothing: the platform sits in the queue that nobody is draining until someone restarts the process by hand (#90386 reports 4h17m of that, with cron unaffected throughout). Call the ensure on the already-queued path as well. It is already idempotent and already cheap: it returns immediately unless the tracked task is done, and it routes through the same on_spawn handle tracking, so a live watcher is never duplicated. The queue entry itself is deliberately left untouched. Re-enqueueing would reset attempts and next_retry, restarting the backoff ladder on every fatal error and hammering a provider that is already refusing the connection.	2026-08-22 15:25:45 +05:30
Kshitij Kapoor	41e29a601e	fix(gateway): clear resume_pending for all claimed ledger rows before any redelivery send ... Follow-up to the salvaged #91986: the per-row clear still left rows the loop had not reached exposed — a slow send ahead of them could hold the loop past the inbound-gate timeout and let _schedule_resume_pending_sessions replay those turns. Clearing every claimed row up front closes the duplicate window; claiming already spent the redelivery attempt, so the ledger retry path is unchanged.	2026-08-22 15:25:30 +05:30
HexLab98	ce944a5a55	fix(gateway): do not let boot-path sends hold the inbound gate ... Restart notification and obligation redelivery ran before the startup-restore gate opened, so one hung Telegram send queued inbound on every platform. Bound those sends with the same timeout the resume gate already uses, and clear resume_pending before send so a timed-out redelivery cannot also replay the turn.	2026-08-22 15:25:30 +05:30
abundantbeing	a2a23a8f7e	fix(clients): hide compaction carriers across surfaces	2026-08-21 21:53:11 +05:30
Ben Barclay	952d1cef18	fix(gateway): prompt-send failures keep their diagnostic detail; clarify caller path gets contract tests ... Two review nits on the clarify ambiguity fix: - _approval_send_outcome swallowed the failure detail the old inline callers logged (scheduling exception text / SendResult error). Both the approval and clarify lanes now share one warning with the detail, logged in the classifier itself. - The clarify tests pinned the disposition helper but nothing proved the ambiguous branch actually reaches the bounded wait. The send-then-wait sequence is extracted to _clarify_send_then_wait (the callback closure now just binds context onto it) and the suite gains caller-path tests: ambiguous/sent -> wait_for_response with the generated clarify_id and configured timeout; definitive failure -> sentinel without waiting; no-response timeout sentinel preserved; plus caplog assertions that failed sends log their detail. Relay + gateway sweep 289/289.	2026-08-20 19:56:01 -07:00
Ben Barclay	a7cfe391f7	fix(gateway): clarify prompt sends get the same ambiguous-timeout handling as approvals ... The clarify caller treated a send-scheduling timeout as a definitive failure: clear_session() + '[clarify prompt could not be delivered]'. Same physics as the approval card fixed earlier in this PR — the card may well have posted with a late connector ack — so the teardown ran out from under a rendered clarify card and the user's answer resolved nothing. New _clarify_send_disposition() routes the outcome through _approval_send_outcome: only a DEFINITIVE failure (error result, non-timeout exception, no future) clears the registration and aborts; ambiguous logs a warning and falls through to wait_for_response, whose existing bounded wait already handles the truly-lost-card case. This makes the boundary rule stated in the ambiguity test docstring hold for the clarify lane, not just approvals. Tests: 5 disposition tests mirroring the approval suite, including clear_session-not-called on timeout. Mutation-verified: folding ambiguous into the failed branch sends test_timeout_keeps_registration_armed_and_proceeds_to_wait red. Relay + gateway sweep 283/283.	2026-08-20 19:56:01 -07:00
Victor Kyriazakos	5210dd48b8	fix(gateway+relay): approval prompts survive ambiguity without duplicates; streamed finals keep block formatting ... Three live findings from rc.4 staging, all on the relay-fronted Slack path, all with the failure observed in live logs before the fix: 1. Approval-send timeout is AMBIGUOUS, not failed (no re-ask). send_exec_approval through the connector can time out with the card already rendered — the connector may ack after the deadline (slow platform API call, transient backpressure, event-loop stall) — and the timeout-as-failure path re-sent and produced duplicate cards. The outcome is now tri-state: sent / failed / ambiguous. Ambiguous = no re-send, no text fallback; the prompt registration stays armed so a late tap still resolves. Only a definite send error falls back to text. 2. pending_approval tool results forbid re-issuing the command. With one card correctly armed, the agent could still mint a SECOND card by re-running a rephrased variant of the gated command after reading the pending_approval tool result (observed live: same command re-issued in a different form, two cards). The tool message now instructs: do not re-run/rephrase; wait or report pending. Applied to both the terminal and execute_code arms. 3. Draft interim AND seal frames carry format_hints. format_hints are stamped on send, edit, and send_for_platform, but both draft-frame builders (send_draft interim + _seal_open_draft seal) shipped bare metadata. A streamed final therefore arrived at the connector hintless and sealed as a plain code block while non-streamed sends rendered native markdown blocks (observed live: language-tagged block on send/edit, downgrade on streamed seal). Both sites now stamp _with_format_hints_for_chat (destination-resolved, same pattern as the existing lanes). Verified live after the fix against the platform's stored message payload: rich_text_preformatted with language field on a streamed seal. Tests: tri-state outcome unit tests (5), draft/seal hint stamping + knobs- off regression control (2, RED-first), existing format-hints suite intact (14/14). Mutation-verified: reverting the adapter hunk sends test_draft_interim_and_seal_frames_carry_hints red; restore -> green. Boundary sweep (text egress lanes crossing the frame contract): send ✓ (pre-existing) edit ✓ (pre-existing) send_for_platform ✓ (pre-existing) draft-interim ✓ (this PR) draft-seal ✓ (this PR); task_card lane carries no text content — exempt.	2026-08-20 19:56:01 -07:00
Ben Barclay	ee000768ce	Merge pull request #90761 from NousResearch/fix/scale-to-zero-cron-aware-idle ... fix(gateway): count cron and API-server work in the scale-to-zero idle predicate	2026-08-21 08:19:36 +10:00
Ben Barclay	0215930526	review: fail-awake work accounting + rename is_idle param to active_work_count ... Address sol-reviewer findings: - The shared shutdown-drain counters swallow exceptions to 0 — fine for a drain, unsafe for a suspend predicate (a transient read failure made live work look idle, reopening the mid-job freeze). The suspend path now reads both sources itself and treats an unreadable source as work (sentinel 1, fail-awake) with a debug log. A MISSING api_server adapter remains a normal not-work state. - is_idle()'s parameter renamed running_agent_count -> active_work_count: it receives the broad aggregate, and the old name invited future callers to pass only agents again. - New failure-path tests: unreadable cron source and unreadable API source each hold the machine awake (both fail against the fail-open shape); missing adapter stays idle-capable.	2026-08-21 07:37:11 +10:00
Chris Fontes	64505a2b83	fix(resolve_turn_limit): gateway bridge null handling, TUI resolver, docs ... Addresses teknium1 sweeper review on PR #67696: 1. Gateway bridge: Skip str(None) bridging when YAML value is Python None (from or bare ). Previously str(None) → None → unlimited instead of default 90. Now clears stale env var so resolver applies default. 2. TUI: Route _cfg_max_turns through resolve_turn_limit instead of bare int(). Old code crashed on none/unlimited and swallowed 0 via . HERMES_TUI_MAX_TURNS env var also routed through resolver. 3. Docs: Document unlimited spellings (none/unlimited/infinite/0/-1) in configuration.md. 4. Tests: Add TestGatewayBridgeNullHandling (4 tests) and TestTUIResolver (8 tests) covering null handling, string spellings, env var override, and legacy root-level config. All 50 tests pass.	2026-08-20 04:50:39 -07:00
Chris Fontes	5046282867	feat(config): resolve_turn_limit — first-class 'none'/'unlimited' for agent.max_turns ... Previously agent.max_turns only accepted positive integers. Setting it to 'none', 'unlimited', or 0 — all natural ways to say 'no limit' — either crashed int() or was silently skipped by `or` checks, falling back to 90. This adds resolve_turn_limit() in hermes_cli/config.py as the single normalization point. It accepts: - int/float → int(raw) (floats truncated) - numeric string ('120') → int(raw) - 'none'/'unlimited'/'infinite'/'∞'/'-1'/'0' (case-insensitive, whitespace-tolerant) → sys.maxsize sentinel - YAML None/null → default (90) - bool/list/dict/garbage → default (with debug log) All config-reading sites (cli.py, gateway/run.py, cron/scheduler.py) now call this instead of bare int(), so agent.max_turns: none in config.yaml becomes a first-class supported spelling of 'unlimited'. The sentinel (sys.maxsize) survives the str()→int() round-trip through the HERMES_MAX_ITERATIONS env-var bridge in gateway/run.py and works in every <, >=, remaining = max - used comparison without requiring call sites to learn about a special value. Includes 38 tests covering the full spelling table, the str→int env-var round-trip, and sentinel properties.	2026-08-20 04:50:39 -07:00
Teknium	95fa814269	feat(process): positive process identity — spawn tags, machine spawn ledger, Windows job-object self-attach ... Every long-lived Hermes process is now positively identifiable so reapers never have to guess lineage from PPID archaeology or cmdline shape: - hermes_cli/process_identity.py (new): HERMES_SPAWN tag build/parse, spawn-ledger.json self-registration keyed on (pid, create_time) — PID reuse cannot forge the pair — with #89298-style corrupt-file quarantine, and a kill-on-close job-object self-attach (BREAKAWAY_OK preserved for the existing CREATE_BREAKAWAY_FROM_JOB escape hatches). - serve/dashboard (web_server.py) and the gateway entry point register themselves at startup and attach to the job; Desktop legacy HERMES_PARENT_PID/winms marker reused as spawner identity so lineage works with every Desktop version. - Desktop stamps HERMES_SPAWN on backend spawns (parent-process-identity.ts). - hermes update gets a positive-identity rung ahead of the heuristic ones: _ledger_reapable_backend_pids reaps holders the ledger PROVES are orphaned backends (purpose reapable + recorded spawner provably dead) in ANY update context. Ledger-unknown holders fall through to the existing rungs. 22 new tests, sabotage-verified.	2026-08-20 04:47:38 -07:00
Ben Barclay	743dc935f5	fix(gateway): count cron and API-server work in the scale-to-zero idle predicate ... _scale_to_zero_is_idle() consumed _running_agent_count(), but cron jobs run through a standalone AIAgent on the scheduler's own thread pool and API-server runs live on the adapter — both outside _running_agents (the same blind spot the #60432 shutdown-drain fix addressed with _active_work_count()). The idle predicate therefore read True DURING a running cron job; a suspend at that moment freezes the job mid-flight. Observed live on staging 2026-08-20: is_idle held True throughout the 10:45:04-22 cron run — only watcher-tick timing (next tick 9s after completion) avoided a mid-job freeze. Use _active_work_count() (agents + cron + API runs). New tests cover a running cron job and an active API run each blocking idle, plus the all-quiet True case; both blocking tests fail without the fix.	2026-08-20 20:53:16 +10:00
pierrenode	a1ddb54840	fix(gateway): tag the loop-liveness and heartbeat-poll tasks as permanent supervised watchers (#84558) ... #84327 excluded _spawn_supervised's permanent watchers (session-expiry, kanban, reconnect, the scale-to-zero watcher itself, ...) from _scale_to_zero_has_live_background_work() via a _hermes_supervised_watcher tag, because counting them made an armed gateway consider itself busy forever and never go dormant. Two more permanent, infinite-loop tasks are added to _background_tasks OUTSIDE _spawn_supervised and were untagged: - _loop_heartbeat_task (loop_heartbeat_forever, #66892): a `while True` loop started unconditionally in start() on every gateway boot. Extracted the inline spawn block into _start_loop_heartbeat_task() so it's independently testable, matching the existing _start_heartbeat_poller() pattern. - _heartbeat_poll_task (_poll_loop in _start_heartbeat_poller): also a `while True` loop, started the first time a session registers a heartbeat watch, and then permanent for the rest of the process. Because _loop_heartbeat_task starts on every boot, it alone made _scale_to_zero_has_live_background_work() return True forever on every armed instance, regardless of the #84327 fix -- confirmed empirically against the real method with the exact untagged-task shape this task has. Two new regression tests spawn each task through its real production entry point and assert the busy check returns False; both fail against the unfixed code (missing method / real assertion failure). Co-authored-by: pierrenode <298902573+pierrenode@users.noreply.github.com> Co-authored-by: Ben Barclay <ben@nousresearch.com>	2026-08-20 20:24:40 +10:00
victor-kyriazakos	3683e70043	feat(relay): live-card ops — native draft streaming + task cards over the relay (gateway half) (#85796) ... * feat(relay): live-card ops — native draft streaming + task cards over the relay (gateway half) NS-658. Three additive ops within contract v1, emitted only when the connector's negotiated descriptor advertises them: {op: draft, chat_id, draft_id, content, final, metadata} {op: task_card, chat_id, card_id, chunks, metadata} {op: task_card_stop, chat_id, card_id, metadata} The gateway side is deliberately dumb: no platform API knowledge, no new config keys. Slack mechanics (chat.startStream/appendStream/stopStream, per-workspace feature-gate cache, send+edit fallback) live connector-side where the platform adapter lives in the relay model. Semantic bridge: base send_draft is Telegram-shaped (draft clears; final is a separate send). Slack native streaming makes the stream THE message. The adapter tracks the open draft per chat and converts the turn-final send() into draft(final=true) so the connector seals the stream instead of posting a duplicate; the stream ts returns as the message identity. A failed frame disarms interception so the edit-based fallback's real send goes through untouched. BEHAVIOR CHANGE (deliberate): relay supports_draft_streaming() now requires the descriptor flag AND the draft op. Flag-only was a latent lie — send_draft inherited NotImplementedError, so a connector setting the flag without the op would have crashed the stream consumer's draft path. supported_ops stays fail-open for legacy (pre-contract) ops; draft/task_card did not exist pre-contract and must not fail open. Task cards ride #85476's adapter-agnostic TurnRunner seam (hasattr on send_native_task_card_progress); supports_native_task_cards() is the descriptor probe. Connector half + E2E harness pair follow in the gg repo. * fix(relay): expose native_task_cards_enabled() on the relay adapter Live-canary finding (Alice, staging): the TurnRunner's task-card lane probes adapter.native_task_cards_enabled() (the native Slack adapter's opt-in contract). The relay adapter only offered supports_native_task_cards(), so the hasattr gate failed silently and tool progress stayed on the text path — draft streaming worked, cards never rendered. Alias it to the descriptor probe. * fix(relay): match task-card methods to the TurnRunner's native keyword contract Live-canary finding #2 (Alice, staging): gateway/run.py's card lane calls send/stop_native_task_card_progress with the NATIVE Slack adapter's signature (tasks/title/reply_to/metadata/fallback_text, keyword-only) — PR 85796's relay methods took a positional card_id, so every call raised TypeError('unexpected keyword argument reply_to') in the progress task, repeatedly killing the card publisher (and the retry loop resent the final delivery 4-5x). Card id now derives per turn thread (turn:<reply_to>), thread_ts anchored like draft; title/fallback_text accepted for parity, not forwarded (plan-mode stream renders chunks). * fix(relay): one draft stream per turn for stream-is-the-message adapters Live-canary finding #4 (Alice, staging): the stream consumer bumps draft_id at every tool boundary so Telegram-shaped drafts animate each text segment as a fresh preview. On relay Slack NATIVE streaming a new draft_id opens a brand-new chat.startStream — the user saw one frozen message per segment (stuck streaming cursor ▉, never sealed: only the LAST stream gets the final=true seal) plus the real final; 5-6 cumulative snapshots per turn. Adapters that mark draft_stream_is_message keep ONE stream per turn: tool progress lives in the native task card, and the connector's suffix-delta falls back to whole-text append on prefix mismatch, so segments append cleanly. Telegram-shaped drafts keep the per-segment bump. * fix(relay): don't seal the native stream at tool boundaries — only the turn-final does Live-canary finding #5 (Alice; supersedes the incomplete #4 which was necessary but not sufficient). Root cause CONFIRMED by integration trace (test_live_cards_flow_trace.py, real consumer semantics + real adapter + stub transport): at every tool boundary the consumer calls _send_or_edit(finalize=True), which skips the draft path and issues a real send(); the relay adapter's seal-interception converts THAT into draft(final=true) — sealing the stream once per segment. Timeline showed 3 seals for a 3-segment turn: exactly the frozen cumulative ▉ snapshots seen live (the replaced stream never gets stopStream, keeping its cursor). Fix: for draft_stream_is_message adapters, a segment-break finalize (finalize=True, is_turn_final=False) stays ON the draft path as another cumulative frame; only got_done (is_turn_final=True) falls through to send() and seals. Telegram-shaped platforms unchanged. Trace test now pins the invariant: ONE user-visible message per turn. * fix(relay): strip the text cursor from native draft frames Live-canary finding #6 (Alice) — the ACTUAL duplicate-content mechanism, confirmed by full-flow scan of both sides' code + logs. The consumer appends its text cursor (▉) to every non-final display_text tick. The connector's stream sender diffs CUMULATIVE frames via prefix check: 'abc▉'.startsWith → 'abc def▉' is NEVER a prefix match (the cursor sits mid-string), so deltaFor falls back to whole-text append on EVERY tick — chat.appendStream stacks each full cumulative snapshot (cursor included) into the ONE stream message. Exactly the observed thread: repeated blocks, each ending in a frozen ▉, growing per tick. Fixes #4/#5 were real (one stream per turn now) but this was the last mechanism standing. Native streams render their own typing indicator, so the text cursor is pure noise on this path: strip it from draft frames. Prefix check now holds; every tick appends only its true suffix delta. * fix(relay): seal-interception covers EVERY egress door, not just send() Live-canary finding #7 (Alice): one duplication remained after #6 — the stream froze mid-word with the live indicator (never sealed) and the final posted as a separate message. Log receipt: 'Queued follow-up: final text delivery confirmed; delivering explicit media before continuing' — the turn's final went out via the DELIVERY RESOLVER lane (gateway/delivery.py), which calls send_for_platform() DIRECTLY, bypassing send() and its seal-interception. The open stream never absorbed the final; it arrived as a plain 'send' op → chat.postMessage. Fix: hoist the open-draft check to the top of send() (ahead of the explicit-platform branch) AND add it to send_for_platform() — an open native stream absorbs the turn-final regardless of which egress door it arrives through. The stream IS the message. * fix(relay): failed seal falls back to plain send (PR 85796 AI-review point 1) A turn-final seal that fails at the transport must never swallow the final answer: the stream consumer has already disabled the draft transport for the run, so a failed _seal_open_draft returning success=False meant the user got NOTHING. Both seal-interception sites (send + send_for_platform) now fall through to the regular plain-send path on seal failure, with a warning receipt. Also mitigates AI-review point 2 (sticky _open_draft_by_chat after an abandoned turn): a stale entry's failed seal no longer blocks the next turn's delivery. * fix(relay): arm seal-interception optimistically; never disarm on ambiguous failure (audit G-D1) Deep-audit defect G-D1 (HIGH): the outbound leg is at-most-once on the wire but its ack channel is lossy — send_outbound timeout (30s) and WS-drop 'failures' frequently mean the frame WAS delivered and the connector stream is open. send_draft popped _open_draft_by_chat on any failure, disarming seal-interception while the connector stream lived: the turn-final went out as a plain send → orphaned mid-word stream + complete duplicate final (intermittent; needs a drop/timeout inside the draft window). Fix: arm the entry BEFORE the transport call and keep it armed on failure/exception. Safe in every case: sealing a non-existent stream opens+seals a single complete message connector-side, and a truly failed seal already falls back to plain send at both interception sites. Stale-entry damage is self-healing (one warning + plain send). * fix(relay): gateway-side sealed-draft tombstone — G-D1 arming must not resurrect sealed streams Regression fix on G-D1 (live: 'worse than before' — escalating frozen prefixes). Optimistic arming had no seal-awareness: a straggler frame arriving AFTER the seal re-armed _open_draft_by_chat for the already- sealed draft_id; the next send was converted to draft(final=true) on the tombstoned connector key, which CLEARED the connector tombstone (final frame = new-turn signal), re-opened a stream with cumulative content, and left it frozen — repeating per straggler: 4-5 escalating frozen snapshots. Mirror the connector: _sealed_draft_by_chat records the sealed draft_id per chat (tombstoned BEFORE the seal's transport call); send_draft for a sealed draft_id is a success no-op (content already in the sealed message) and never arms. A new turn's fresh draft_id arms normally. * fix(relay): key stream/card state per (chat, turn anchor) — parallel turns must not collide (finding #10) Live finding #10 (Alice; three concurrent turns in one flat DM): all coordination state was keyed per CHAT on a one-active-turn assumption. Three parallel turns produced: turn B's task card merged into turn A's (both were card 'turn:root' — reply_to is None in flat DMs), B left cardless, and _open/_sealed_draft_by_chat clobbered across writers (3x duplicate finals on the last turn). Per-turn machinery was correct; the keys were not. Fix: _draft_key(chat, metadata) = chat + the turn's thread anchor (inbound stamps thread_ts = event.thread_ts or ts on every top-level message, so each turn has one even in flat DMs). draft arming, seal tombstones, both interception sites, and the task-card id all derive from the same anchor. New trace test pins two interleaved turns: distinct cards, own-stream seals, no leaked plain send, no cross-turn tombstone drops (289 tests green). * fix(gateway): preserve cumulative native stream across tools * fix(gateway): consumer-declared final — the seal carries the true final Three composed fixes for the Slack live-cards duplicate-final class: 1. finish(final_text): TurnRunner passes the completed final_response (verifier footer, completion explainer included) as the authoritative finalize payload. The native-stream seal delivers the TRUE final, so post-stream mutation no longer forks a corrective plain send (#11). 2. Interim-send contract: commentary and segment-tail sends carry a gateway-internal _interim_send marker; relay seal-interception skips them at both egress doors. A mid-turn interim send can no longer seal the live stream and orphan the real final into a duplicate. 3. Queued-follow-up lane reconciles an unconfirmed final by EDITING the consumer's delivered message in place (sealed stream = regular message, chat.update live-verified); plain send only as fallback. This was the actual duplicate lane in the parallel canaries — every duplicated turn logged 'final stream delivery not confirmed; sending first response' (subagent-completion queued inbound), not parallelism. Also: draft frames stay prefix-stable gateway-side (no fence-closing, no segment state reset, no commentary reset for stream-is-the-message adapters; MagicMock-safe 'is True' guards). * test+docs: streaming-contract coverage completeness + maintenance guidelines Coverage: two gaps closed on the consumer-declared-final contract — (1) send_for_platform (the delivery-resolver egress door) honors the _interim_send contract: no seal, marker stripped before the wire; (2) finish(final_text) on a turn that never streamed does not adopt the final (delivery ownership stays with the gateway's normal send path for non-streaming models / tool-only turns). Docs: AGENTS.md 'Known Pitfalls' gains the streaming delivery contract — the four invariants of stream-is-the-message adapters (prefix-stable frames, consumer-declared final, interim-send marker, reconcile-by-edit), each traced to its live incident, plus the live-probed Slack streaming API ground truth and the MagicMock 'is True' guard-style note. * fix(relay): seal transport failure must never silently lose the final (review B1) Two halves of one silent-loss path, live-probed on the review branch: 1. adapter: _seal_open_draft did not catch transport exceptions. A socket drop at seal time raised out of send(), skipping the fail-open plain send entirely. Now: retry the SAME idempotent final frame once (the connector's sealed-key tombstone returns the original stream ts for a repeated final — a retry can never open a second stream or duplicate), then report failure so the caller's fail-open path runs. 2. consumer: the turn-final retry (elif not _already_sent) called _send_or_edit with finalize=False, which re-entered the DRAFT-FRAME branch. Its no-op dedupe compared the adopted final against the last unsealed frame, matched, and returned True with ZERO transport calls — final_response_sent went green, delivered_final_matches reconciled, the gateway suppressed its fallback, and the user never received the answer. finalize=True keeps this retry out of the draft branch. Regression suite: tests/gateway/test_relay_seal_failure.py (3 tests). Mutation evidence in follow-up verification: reverting either half sends the suite red. * fix(relay): draft ids unique across gateway incarnations (review B3) The relay connector tombstones sealed streams by (channel, draft_id) and keeps up to 512 of them; they outlive the gateway process. Relay gateways are disposable BY DESIGN (scale-to-zero), and _draft_id_counter restarted at zero every incarnation — so the first turns after every scale-from-zero in a recently-active channel replayed already-sealed wire identities. The connector answered those frames straight out of the old tombstone: zero Slack API calls, the OLD message ts returned as the new turn's identity, the new answer silently dropped while gateway-side flags recorded success. Seed the counter from wall-clock milliseconds at process start. Ids stay plain ints within the existing contract op; incarnations cannot overlap for realistic turn counts and restart gaps. Regression: tests/gateway/test_draft_id_restart_uniqueness.py — the seed test fails on the old code (seed 0 is not epoch-scale). * fix(relay): stream/card state keyed per TURN, not per thread anchor (review B2) The thread anchor is the wrong coordination identity — simultaneously: - too coarse: two parallel turns replying INSIDE ONE Slack thread share thread_ts. Live-probed on the review branch: turn A's final sealed turn B's stream with A's content while A's own stream stayed open, and B's final degraded to a plain send. - too fragile: a flat DM with no thread metadata degraded to the bare chat id, re-creating the original finding-#10 collision the anchor was meant to fix. _draft_key now prefers the triggering inbound message id (message_id / reply_to_message_id — per-turn by construction; the gateway's Slack thread metadata and the consumer's send path both stamp it), falling back to the thread anchor, then the bare chat. The consumer stamps the same reply_to_message_id on draft frames so frames and the turn-final resolve to one key. Task-card ids share the derivation via _card_key (one helper for send AND stop, so the stop always hits the stream the send opened). Legacy resolver-lane callers with placement-only metadata still seal via _match_open_draft's fallback — but ONLY when exactly one stream is open. With several open, an identity-less send stays a plain send: a duplicate message is recoverable, sealing someone else's stream is not. Regression: tests/gateway/relay/test_relay_turn_keying.py (7 tests). * fix(relay): stream-is-the-message is a Slack semantic, gate it on the descriptor (review B4) draft_stream_is_message was hardcoded True on the relay adapter class, i.e. for EVERY relay platform. The base send_draft contract is Telegram-shaped — the draft clears client-side and the final arrives as a separate real send that becomes the history message. With the flag forced on, any non-Slack connector advertising the draft op had its turn-final intercepted into draft(final=true): probed on the review branch with a telegram descriptor, the op stream was [draft(final=false), draft(final=true)] and NO send — no history message would ever be posted. Gate the flag on the negotiated descriptor platform (slack), and skip arming seal-interception entirely when it is off. A future platform with genuine stream-is-the-message native streaming should advertise it via the descriptor rather than widening the platform check by guesswork. Regression: tests/gateway/relay/test_relay_stream_semantics_gating.py (4 tests: gating both ways, telegram final is a real send, slack final still seals). * fix(gateway): mark every mid-turn status lane interim — heartbeats must not seal the stream (review B5) Seal-interception treats the first unmarked send to an armed (chat, turn) key as the turn-final. The consumer's own interim lanes (commentary, tail flush) carry _interim_send, but four gateway-side lanes that fire DURING a streaming turn did not: - long-running heartbeat (default every 180s — probed live: at 3 minutes it sealed the live stream with '⏳ Working — 3 min', the real final posted as a duplicate, and later frames were silently swallowed by the seal tombstone) - inactivity warning - plain-text approval fallback (button lane failed) - background-review notice Add _interim_metadata() beside _non_conversational_metadata and wrap all four call sites. The marker is gateway-internal; the relay adapter strips it before the wire (existing behavior, pinned by test). Note for follow-up: the opt-out shape remains fragile — any FUTURE unmarked mid-turn send lane re-creates this bug. Inverting the contract (explicitly mark the one turn-final send) is the durable fix but touches every adapter's final-delivery path; deliberately kept out of this review-fix series. Regression: tests/gateway/test_interim_send_lanes.py (4 tests). * fix(gateway): interrupted/incomplete turns must not adopt the diagnostic as the stream final (review B6) The finish(final_text) adoption gate checked only 'not failed', but the interrupt/abort returns in agent/conversation_loop.py are {completed: False, interrupted: True, final_response: 'Operation interrupted during …'} with NO failed key. Adopting that diagnostic: 1. sealed the user's streamed partial answer over with the interrupt text (stream-is-the-message: the seal rewrites the whole message), and 2. recorded the diagnostic as the turn-final payload, so delivered_final_matches reconciled and the gateway suppressed its own error-delivery path — the diagnostic became the ONLY thing delivered. Enumerated all 27 final_response-bearing return shapes in conversation_loop.py: every non-happy-path shape carries completed: False (several with a diagnostic final_response and neither failed nor interrupted — retry exhaustion, truncation, codex-incomplete); the happy path routes through turn_finalizer.finalize_turn (completed=True). Gate is therefore: not failed AND not interrupted AND completed is not False. Results lacking the completed key entirely (older callers/test doubles) keep the previous behavior. Regression: tests/gateway/test_stream_final_adoption_gate.py (6 tests, incl. a source-level pin on the run.py call site). * fix(relay): task-card transport failures degrade to failed SendResults (review B7) send_native_task_card_progress and stop_native_task_card_progress let transport exceptions escape. The stop runs inside the progress loop's finally block on the turn-cleanup path, and the post-cancel awaits in gateway/run.py caught only CancelledError — a socket drop during a card publish/stop therefore aborted cleanup BEFORE the final-delivery bookkeeping ran. Three layers, outermost defends any adapter: - both adapter methods catch transport exceptions and return failed SendResults (progress is advisory; the TurnRunner's text fallback already handles failure results) - the progress loop's finally wraps the stop (best-effort; the connector seals orphaned card streams on its own via recycling/eviction) - the cleanup awaits log-and-continue on non-cancellation errors so final-delivery bookkeeping always runs Regression: tests/gateway/relay/test_relay_task_card_failures.py. * fix(relay): a dying turn seals its native stream instead of orphaning it (review B8) Stale-generation exits (/new, /stop mid-stream) and cancellations returned from the consumer's run() with the native stream still open: - the Slack message kept its live streaming indicator forever (the cancellation best-effort edit only runs when _message_id exists, and the native draft path deliberately keeps it None); - the adapter's armed interception state survived the turn, so the next turn on the same key could inherit it and seal a dead draft_id. New adapter op abandon_open_draft(chat, content): seals in place with the text already on screen (the consumer passes its last delivered frame) — the seal adds nothing and claims nothing; delivery flags are never set, so the gateway's normal paths still own whatever happens next. Best-effort by contract (failure reported, never raised); the connector reaps truly orphaned streams via recycling/eviction. The consumer calls it from both death paths: the stale-generation early return and the CancelledError handler. Regression: tests/gateway/test_stream_abandon_on_turn_death.py (4 tests, incl. the next-turn-inheritance hazard). * fix(relay): bound the draft/seal coordination dicts (review M1) _sealed_draft_by_chat's key embeds a per-turn identity, so every completed turn wrote a permanent entry — unbounded growth for the life of a long-running gateway process (the docstring said 'one entry per chat', which stopped being true when the key gained the turn anchor). _open_draft_by_chat could grow the same way via abandoned entries. FIFO-evict both at 512 entries — the same idiom as the sibling bounded cache (_auto_thread_by_chat, capped at 256) and the same size as the connector's own tombstone store. The straggler window the tombstone exists for is seconds long; FIFO is more than enough. Regression: tests/gateway/relay/test_relay_state_bounds.py. * fix(relay): explicit connector rejection disarms interception; exceptions stay armed (review P3) The G-D1 optimistic-arming change silently dropped disarm-on-failure entirely: after an EXPLICIT connector rejection (success=False result — not a transport ambiguity), interception stayed armed even though the stream consumer disables the draft transport on that failure and falls back to edit-based streaming. Its turn-final would then be converted into a seal on a stream the connector just told us is unusable. test_draft_failure_result_propagates claimed to cover this ('must NOT leave seal-interception armed') but passed for an unrelated reason: the stub's canned failure also failed the SEAL, whose fail-open path did the plain send. Split the two semantics and pin each honestly: - explicit rejection (result success=False): disarm — turn-final is a real send (test_draft_failure_result_propagates, now testing what its comment says) - transport exception: ambiguous, stay armed — turn-final still seals (test_draft_transport_exception_keeps_interception_armed, the G-D1 contract) Also corrects commit ba3a24a's claim ('a failed frame disarms interception so the edit-based fallback's real send goes through untouched') to hold again for the rejection case it described. * fix(relay): lost acks are ambiguous, not rejections — on the RESULT channel too (review r2, finding 1) The production ws transport does not raise on ack timeout — it returns {"success": False, "error": "relay outbound timed out"}. The round-1 ambiguity handling keyed entirely on the exception channel, so the shape production actually produces was misclassified as a definite connector rejection. Probed on the head: - lost SEAL ack: skipped the idempotent retry, fell straight to a plain send — duplicate final whenever the seal had actually applied; - lost FRAME ack: the round-1 disarm-on-rejection fired — interception disarmed, frozen native stream beside a plain final. This re-created the original G-D1 ambiguous-ack defect on the result channel. Contract now spans both channels: - transport: the ack-timeout branch tags ambiguous=True. The fail-fast branches (closing / not connected) never sent anything and stay unmarked — they are definite non-delivery. - adapter frame path: ambiguous results keep interception armed (same as exceptions); only definite rejections disarm. - adapter seal path: one shared _attempt() classifier — exception and ambiguous result both mean "unknown"; the SAME idempotent frame is retried once (connector tombstone returns the original stream ts for a repeated final). Only after both attempts stay ambiguous does the caller's fail-open plain send run: a possible duplicate after double ack loss beats a silent loss, and double ack loss on one socket almost always means the transport is down for the plain send too. Regression: tests/gateway/relay/test_relay_ack_ambiguity.py (6 tests, incl. a source-of-truth check that the transport tags the timeout branch and leaves fail-fast branches unmarked). * fix(relay): stream semantics + draft capability resolve per CHAT, not per primary (review r2, finding 2) One RelayAdapter fronts N platforms (Phase 1.5): descriptors accumulate per platform on the transport and egress is tagged per chat — but the round-1 gate keyed draft_stream_is_message and supports_draft_streaming() off the PRIMARY scalar descriptor. Probed on the head: - Slack primary + Telegram chat: the Telegram chat's turn-final was intercepted into draft(final=true) — no real Telegram history message; - Telegram primary + Slack chat: the Slack chat was denied native streaming entirely. Resolve both through _descriptor_for_chat — the same per-chat machinery max_message_length already uses (added for the identical class of bug: the primary's 39000-char cap over-sending into Discord 400s): - new stream_is_message_for_chat(chat_id) on the adapter; arming and NotImplementedError gating use it. The class attribute remains as the single-platform value and legacy-probe fallback. - supports_draft_streaming() gains an optional chat_id kwarg (base signature updated; single-platform adapters ignore it). The consumer passes chat_id with a TypeError fallback for out-of-tree adapters. - the consumer's four draft_stream_is_message reads collapse into one _stream_is_message() helper that prefers the per-chat probe (class-resolved, MagicMock-safe) over the attribute. Platform-name inference ("slack") stays deliberate: a descriptor-level semantic field is the right eventual contract but is a cross-repo wire change — noted for the gg follow-up so future platforms advertise the semantic explicitly. Regression: tests/gateway/relay/test_relay_multiplatform_semantics.py (5 tests: both starvation directions, scalar fallback, per-chat capability gate). * fix(gateway): split delivery + authoritative footer reconciles by suffix, not full resend (review r2, finding 3) The _FINAL_TEXT adoption guard refuses wholesale adoption on split turns — correct (#78541: sealed heads would repeat inside the tail) but it was absolute: a post-split verifier footer never entered the ledger, delivered_final_matches() reported a mismatch, and the gateway resent the ENTIRE body+footer after the split chunks (the #11 duplicate class, one level up). When the authoritative final strictly prefix-extends the split ledger, the missing suffix is the only undelivered content: append it to the live tail and the ledger, so the finalize carries it and the recorded payload reconciles. Non-prefix rewrites keep the full-resend fallback — a rewrite cannot be patched onto sealed heads. Regression: tests/gateway/test_split_final_suffix_reconcile.py (3 tests: suffix rides the tail + reconciles, rewrite still mismatches, unsplit adoption unchanged). * fix(relay): cancellation mid-seal restores open state so abandon can close the stream (review r2, finding 4) _seal_open_draft pops the open entry and writes the local tombstone BEFORE awaiting transport I/O — correct ordering for the straggler race, but CancelledError is not an Exception: a cancel during the await bypassed all failure handling, leaving the remote stream live (visible streaming indicator until connector eviction) while the local state said 'nothing open'. The consumer's abandon pass — added for exactly this turn-death case — found nothing to close and no-oped. On CancelledError: restore the open entry, drop the premature tombstone (only if it is still ours), re-raise. The abandon path then seals the stream in place with the on-screen text. Regression: tests/gateway/relay/test_relay_seal_cancellation.py (2 tests: state restoration, and end-to-end cancel→abandon→remote seal). * fix(relay): thread anchors are placement, not turn identity — revive the placement-only fallback (review r2, finding 5) _match_open_draft's single-open-stream fallback was dead for its primary intended callers: metadata carrying thread_ts/thread_id (placement-only resolver lanes) was classified as having 'turn identity', so those sends never reached the fallback — probed: a plain final posted beside the still-open turn-keyed stream. Only per-turn MESSAGE ids are identity now. Thread-anchored and bare callers share the fallback: absorb into the chat's open stream when EXACTLY one is open; stay a plain send when several are (duplicate is recoverable, wrong-stream seal is not). Callers WITH a message id whose key misses never fall back — their identity is authoritative and a miss means the stream belongs to a different turn. Regression: 4 new tests in test_relay_turn_keying.py (thread-anchored seal, both ambiguous-stay-plain shapes, id-mismatch never steals). * fix(relay): random process nonce for draft-id seeding (review r2, follow-up 6) The epoch-millisecond seed (round-1 B3 fix) mitigates the restart-replay class but is not a uniqueness guarantee: two gateways starting in the same millisecond, a forked process inheriting the class state, or a clock step backwards can all mint colliding wire identities against the connector's per-(channel, draft_id) tombstone store. Seed from secrets.randbits(49) instead: collision probability negligible, no clock dependence, and ids + realistic per-process turn counts stay comfortably inside the connector's JS number range (draft_id?: number, 2^53). Regression test now spawns two real interpreters and asserts their seeds differ — the exact scale-to-zero restart shape, and both start within the same second so a clock-locked seed would fail it. * fix(relay): stamp per-turn Slack egress identity — cache is fallback only (R3-5) The connector (gateway-gateway#210) fills chat.startStream's recipient_user_id / recipient_team_id — required by Slack when streaming to a channel — from metadata.user_id / metadata.scope_id. The gateway stamped only slack_team_id per-turn and left user_id (and scope_id) to RelayAdapter._with_scope, whose per-chat caches are keyed on chat_id alone and overwritten by every inbound message: with users U1 and U2 running overlapping turns in one channel, U2's arrival overwrote the cache before U1's stream opened, and U1's stream carried U2 as recipient_user_id. _thread_metadata_for_source now stamps scope_id and user_id from the turn's OWN source (setdefault — explicit values win), so identity is turn-scoped data on the wire. _with_scope is unchanged and fill-only: the caches keep serving restart/synthetic sends that carry no per-turn identity, which is all they were ever safe for. Mutation evidence: reverting the run.py hunk sends test_thread_metadata_stamps_per_turn_user_and_scope and test_concurrent_turns_carry_their_own_identity red; restore returns green. The _with_scope fill-only tests pass on both trees (existing correct behavior, now pinned against regression). --------- Co-authored-by: Ben Barclay <ben@nousresearch.com>	2026-08-20 15:31:21 +10:00
69k4xmdfm2-blip	21260c3281	fix(gateway): carry the profile in adapter-derived session keys (#88404) ... Adapter ingress derives a session key BEFORE the runner stamps source.profile in _make_profile_message_handler, so the namespace fell back to the active profile and every bot in a multiplexed gateway produced agent:main:<platform>:<chat>. A Telegram private chat reports the user's own id as chat.id, identical for every bot, so two profiles sharing one human collapsed onto a single lane: _pending_text_batches, _active_sessions, the busy-session guard and _post_delivery_callbacks are all keyed on that string. A day of production logs across two bots shows 60 flushes, none carrying the secondary profile's namespace. set_owner_profile records credential ownership on the adapter and _session_key_profile resolves the namespace as source.profile -> _owner_profile -> the session store's resolver, so a secondary adapter keys into its own namespace even before the source is stamped. Stamped sources keep priority, so relay/connector ingress, which routes per event rather than per credential, is unchanged. _configure_profile_adapter installs the owner alongside the other handlers, covering startup and reconnect. Every candidate is type-checked as a non-blank str, and every attribute read goes through getattr: adapters are routinely built without BasePlatformAdapter.__init__, and a duck-typed session store returns a truthy non-string that would otherwise be interpolated into the key as agent:<MagicMock ...>:. Also routes the four call sites that passed no profile at all (feishu media batches, raft, slack _session_key_for_source, telegram photo batches) through the same resolver. test_multiplex_busy_input_mode's secondary-adapter busy case seeded _active_sessions with the unstamped agent:main: key, asserting the pre-fix collapse. It now seeds the lane the profile-owned adapter actually derives. A primary adapter has no owner and an unstamped source, so it resolves exactly as before; with multiplex_profiles off the resolver returns None and every key is byte-identical to today's.	2026-08-19 02:00:16 -07:00
Teknium	c69b6471e6	fix(gateway): real user text stays clean in the transcript on resume-pending turns (#86580) ... The salvaged fix persisted the recovery note unconditionally — correct for the synthesized empty auto-resume turn, but a user who typed real text while resume was pending would get the [System note: ...] scaffold persisted as their own words, leaking scaffolding into the durable transcript (the same class as #81841 on the assistant side). _prepare_resume_pending_message now persists the note only when the original message is blank; real text persists verbatim while the model still receives the wrapped note. Whitespace-only counts as blank. Tests cover all three shapes.	2026-08-18 16:44:32 -07:00
andyst-dev	0cc26777bb	fix(gateway): persist resume recovery notes ... Fixes #86580	2026-08-18 16:44:32 -07:00
ethernet	6e67841a9a	refactor(goals): share one dropped-write warning across managers ... Review fixups for #88965. The goal, loop, and heartbeat managers each had a copy of the same WARNING text. The shared _warn_dropped_write helper in goals.py keeps the three logs identical and greppable as one bug class. The _warm_goals_session_db parameter is now label. The old name ctx said context, but the value is a log label.	2026-08-18 16:08:29 -07:00
ethernet	46d8cf0be3	fix(gateway): /loop paths warm the SessionDB cache off-loop ... The loops delegation (previous commit) moved /loop onto the shared bootstrap windows, but the loop-class gateway callers still constructed LoopManager on the loop thread with no warm-up — the same false-ack class as /goal, one sibling over: - The /loop command handler: a cold init past the window made save_loop discard the write while the reply claimed the loop was set. Reproduced with a 2s init: "↻ Loop set" with nothing persisted; with the warm-up the loop persists. - _post_turn_loop_completion: a cold cache at the turn boundary stalled the loop for the init duration and could drop the tick-completion write. - _loop_wakeup_watcher: the scan reads every persisted loop, so a cold cache ran the state.db init on the loop thread before the first read. All three now warm the cache off-loop first (same helper, same reason as the goal paths). save_loop also logs at WARNING when it drops a write, matching save_goal: the reply has already told the user the loop was set. Review findings (PR #88965): the goal and heartbeat command paths warmed off-loop, the loop-class paths did not.	2026-08-18 16:08:29 -07:00
ethernet	246477a809	fix(gateway): /goal no longer lies when state.db init is slow ... A fresh state.db init (schema DDL, FTS tables, first config import) measures ~300ms warm on a fast machine. The gateway constructs GoalManager on the event-loop thread, and a cold cache ran that init behind a 0.25s bootstrap grace window: on a slow CI box the /goal set path's waits expired and save_goal silently no-oped — the reply said "Goal set (7-turn budget)..." but nothing persisted, and a fresh GoalManager read back no state (first assertion passes, second fails). Two changes, one per caller shape: - Async callers (_get_goal_manager_for_event, _get_heartbeat_manager_for_event, _post_turn_goal_continuation, and the heartbeat poller) warm the SessionDB cache off-loop through the context-preserving executor before constructing the manager (shared _warm_goals_session_db helper). The loop never blocks and the first write lands at any init duration. A bare to_thread would lose the per-turn profile home override under multiplex; the executor hop keeps it (same pattern as the goal judge path). - Sync callers (heartbeat persistence, _goal_still_active_for_session) cannot await, so the bootstrap windows stay: the call that starts the bootstrap waits a one-time init window (1.5s) instead of the short per-call window (0.25s), giving healthy cold inits room to land while a contended migration still degrades to None with only a bounded one-time stall. The bootstrap thread binds the caller's home as a contextvar override so a multiplexed worker cannot cache the default profile's DB under another profile's key. save_goal and heartbeat save_state now log at WARNING when they drop a write, because the reply has already told the user the state was set. Regression test pins the contract: init past the window, write persists, loop gap under 2s (the flake-policy floor for wall-clock bounds; the slow-init margin grew to match, so the test still tells on-loop from off-loop). Independent diagnosis + measurement by jackulau (#88965 review); the off-loop warm-up shape follows their harness table. Simplify-code review (4-agent) contributed the helper extraction and the poller warm-up.	2026-08-18 16:08:29 -07:00
Teknium	c91681c69a	fix(tools): large tool results persist to HERMES_HOME/cache/spillover instead of truncating when no sandbox env is active ... Sessions that never ran a terminal command (MCP-only, cron, gateway) have no active sandbox environment, so maybe_persist_tool_result() got env=None and fell through to the inline-truncate fallback -- a 467K MCP result was cut to a ~1.3K preview with no file written ('Full output could not be saved to sandbox'). Now the host-side cases (env=None or the local backend) write the spill file directly to $HERMES_HOME/cache/spillover/<id>.txt, alongside the other Hermes-owned caches instead of littering /tmp. Remote backends (docker/ssh/modal/daytona) keep the in-sandbox env.execute() write since read_file resolves in-sandbox there. Cleanup: the gateway housekeeping loop prunes spillover hourly with the other media caches, and a once-per-process best-effort prune on first spill covers CLI-only installs.	2026-08-18 02:12:13 -07:00
Victor Kyriazakos	22f0f22298	fix(cron): manual runs no longer silently drop media attachments ... Field report (enterprise, v0.20.0): cron jobs delivering text + PDF/image attachments to Slack DMs deliver both on scheduled ticks but text-only on manual `hermes cron run <job-id>`. Same box, same token, same scopes — the divergence is process context and error visibility, not credentials. Three defects, one bug class (attachment failures invisible + policy divergence between the gateway process and standalone processes): 1. Standalone lane swallowed warnings: platform standalone senders (Slack files_upload_v2, Discord, ...) report per-file upload failures in result['warnings'] while returning success=True for the delivered text leg. _deliver_result only read result['error'], so the run was marked ok and the attachment vanished without a trace. Warnings now surface into delivery_errors (and the job's last_error). 2. Live-adapter lane swallowed media failures: _send_media_via_adapter logged failures at WARNING and returned None. It now returns per-file error strings and _deliver_result records them — text-delivered-but- attachment-failed is a visible partial failure on BOTH lanes. 3. Media-policy env bridge was gateway-only: gateway.strict / media_delivery_allow_dirs / trust_recent_files were translated from config.yaml to the env vars validate_media_delivery_path reads ONLY in gateway startup. A CLI-process manual run filtered attachment paths under a different policy — in strict/allowlisted deployments the exact reported symptom (scheduled delivers, manual drops, silently). The translation now lives in gateway/media_policy.apply_media_policy_env (idempotent, env-wins, never raises); gateway startup delegates to it and _deliver_result applies it before filtering. Attachments dropped by the policy filter are also reported in the run status instead of only a stderr WARNING. On v0.20.0 specifically the failure was double-blind: the pre-9cf2cbd382 isinstance(resp, dict) gates meant upload failures were undetectable in the sender AND unsurfaced by the scheduler. 9cf2cbd382 (in 2026.8.13) fixed detection; this fixes visibility and policy parity. 8 new tests (tests/cron/test_media_delivery_parity.py): warnings→errors, clean-delivery control, media-reaches-sender control, live-adapter failure/dropped-path reporting, bridge helper semantics, strict+allowlist end-to-end in a non-gateway process, and the .env-strict/config-allowlist split that reproduces the field symptom. Mutation check: disabling the warnings loop and the bridge fails exactly the 2 guarding tests.	2026-08-17 17:19:50 -07:00
Teknium	6f2dbf3a05	fix(gateway): widen scope-aware session DB resolution to the runner and release cached handles ... Follow-up to the salvaged #88632 (Jack Lau) fix for #88532, extending the same repair to the sibling frozen-at-init handle and closing the handle lifecycle gap the per-path cache introduces. 1. GatewayRunner._session_db had the identical bug class: bound once as AsyncSessionDB(SessionDB()) in __init__ on the root home, while /resume, /title, /history and session search all execute inside _profile_runtime_scope on a multiplexed gateway. Convert it to the same property-with-pin pattern: per-access resolution of _default_db_path(), one cached AsyncSessionDB per resolved path under a lock, and assignment preserved as an explicit pin (many suites install fakes or None). Construction-time priming keeps the #88235 init-failure broadcast at startup. 2. Handle lifecycle: the per-path caches accumulate one open SessionDB per profile served, but the shutdown path closed only store._db / runner._session_db - which now resolve just the shutdown task's own (root) scope. Secondary profiles' handles would strand their WAL write locks until process exit, recreating the abandoned-handle leak b454e4da76 fixed and breaking --replace restarts with 'database is locked'. Add close_all_db_handles() / close_all_session_db_handles() sweeps and call both from the gateway teardown path. SessionDB.close() is idempotent, so the root handle being closed by both the legacy loop and the sweep is safe. Tests: sweep coverage plus a runner-property scope/pin/cache test in tests/gateway/test_multiplex_session_db_profile_scope.py.	2026-08-17 16:34:10 -07:00
Gille	68a713385e	fix(gateway): preserve configured context window	2026-08-17 15:46:23 -07:00
Teknium	481156139d	feat: misfire catch-up for external cron providers ... When an external scheduler (Chronos on hosted deployments) cannot deliver a fire — dead loopback hop at fire time, retry budget exhausted — the job's next_run_at stays parked in the past and nothing ever runs it: external providers have no local tick loop, so the day is silently lost even if the gateway heals minutes later (4 consecutive nightly misses in the field). fire_overdue_jobs() in cron/scheduler_provider.py, called from the gateway housekeeping loop every 5 minutes: - No-op for the built-in ticker (its tick loop already self-heals past-due jobs) and when cron.misfire_grace_minutes <= 0. - Waits out a grace window (default 10 min) so the external scheduler's own retry backoff gets first right to deliver. - Claims via the provider's claim_fire (store CAS — a concurrent late external retry is de-duplicated) and runs fire_claimed in a daemon thread, mirroring the webhook admission pattern, so housekeeping never blocks for the length of an agent run. Provider re-arm logic (Chronos NAS one-shots) runs exactly as for a normal fire. Docs: cron.md section + cron.misfire_grace_minutes reference.	2026-08-17 11:42:25 -07:00
Teknium	cb1b1da219	fix: surface missed cron fires as last_fire_error on the job record ... On hosted deployments a scheduled fire that cannot be forwarded to the gateway api_server (dead 8642 listener, gateway down) was invisible outside gui.log: no execution row is created because the claim never happens, so `cronjob list` showed a healthy job that silently missed days of scheduled runs (4 consecutive nightly misses in the field, diagnosed only by log grep). Changes: - cron/jobs.py: note_fire_forward_failure() durably stamps last_fire_error ({at, detail}) on the job record; mark_job_run clears it on the next successful run so it always describes current auto-fire health (mirrors preflight_alerted/drift_alerted). - hermes_cli/web_routers/cron.py: the dashboard fire webhook stamps the job on the gateway-unreachable path, best-effort (never disturbs the 503/Retry-After retry contract or the OOF-266 intentional-stop drop). - tools/cronjob_tools.py: _format_job carries last_fire_error so the agent-facing cronjob list surfaces it. - hermes_cli/cron.py: `hermes cron list` prints a red "Missed scheduled fire" line. - web/: dashboard CronPage renders the miss; api.ts type updated. - gateway/run.py: one-time startup warning when an external cron provider is active but the api_server adapter is not running (the fire path is dead-on-arrival; most common cause is API_SERVER_KEY missing from an unsupervised gateway relaunch). - website/docs: cron doc section on missed fires.	2026-08-17 11:29:10 -07:00
kshitij	979ca57a50	Merge pull request #88244 from kshitijk4poor/fix/handoff-cleanup-race ... fix: prevent handoff leg data loss + surface state.db corruption to users	2026-08-17 17:41:44 +05:30
kshitij	59f302fef9	fix: prevent handoff leg data loss + surface state.db corruption to users ... Two data-loss bugs reported by users: 1. /handoff CLI→gateway race (#88234): After /handoff completed, CLI cleanup called finalize_session on the session the gateway just reopened. This set end_reason on a row the gateway was actively writing to, causing the handoff leg to vanish from session history and breaking session_search recall. Fix: add _handed_off_session_ids module-level set (mirrors _single_query_finalize_attempted_session_ids pattern). _handle_handoff_command registers the session_id on completion; _should_emit_cleanup_session_finalize and _emit_interrupted_session_end check it before firing. 2. state.db corruption silent failure (#88235): When SessionDB init failed at gateway startup, the error stayed in logs — messages flowed but nothing was persisted, with no user-visible indication. Fix: store _session_db_init_error on GatewayRunner, broadcast a recovery-guidance message to all home channels via _send_session_db_warning_notifications() after the gateway connects. Also improved the 'corrupt' persistence cause wording in _format_turn_completion_explanation to include the full recovery path (hermes doctor --fix, sqlite3 .recover, backups). Tests: 6 new tests for handoff cleanup race, 3 for corruption wording. All existing CLI/turn-completion tests pass.	2026-08-17 13:19:58 +05:30
Shannon Sands	f755ed5e90	fix(gateway): surface multiplex profile failures (OOF-3)	2026-08-16 20:32:09 -07:00
Adolanium	fccf2b718e	fix(gateway): complete /loop ticks after streamed already_sent turns ... Streamed replies return None so the adapter does not send twice. The /loop hook then saw empty text and never ran, so awaiting_response stayed true and later ticks never fired. Stash the delivered text on the event and use it for the post-turn hooks. /goal uses the same path. Tests: tests/gateway/test_loop_command.py	2026-08-16 06:31:07 -07:00
Merge_Conflict - Pasi	309cf2c5e2	fix: honor JSON-array string forms for skills.disabled and agent.disabled_toolsets ... `hermes config set` and JSON-mode editor saves store lists as quoted strings (e.g. '["skill-a","skill-b"]' or "['memory']"). Both disable filters treated such a string as a single name, so curated disable lists silently filtered nothing with zero diagnostics. Add parse_config_string_list() in agent.skill_utils and use it in _normalize_string_set (skills.disabled / platform_disabled) and at every agent.disabled_toolsets read site: tools_config resolve + reconcile, CLI, gateway agent construction (both sites), cron scheduler, and prompt_size. A scalar string still names a single entry (#13026); malformed JSON falls back to the single-name behavior instead of raising. Fixes #86661	2026-08-16 06:24:56 -07:00
fangliquanflq	b3df990832	fix(gateway): offload hygiene streak persistence	2026-08-16 02:02:06 -07:00
fangliquanflq	39d2d858fd	fix(gateway): persist hygiene failure cooldown rung	2026-08-16 02:02:06 -07:00
Christopher	3a61fa86fa	fix(gateway): keep broad connection phrases out of the provider-error gate ... Fixes #86570	2026-08-16 01:50:44 -07:00
Christopher	5cfa2b2f0f	fix(gateway): surface actionable message for local model server connection errors	2026-08-16 01:50:44 -07:00
fangliquanflq	faaf2ae093	fix(gateway): read routed profile model config	2026-08-16 01:49:53 -07:00
fangliquanflq	9219cd3944	fix(gateway): isolate post-turn loop failures	2026-08-16 01:39:14 -07:00
fangliquanflq	2f6bbfbcbc	fix(gateway): release loop ticks after empty responses	2026-08-16 01:39:14 -07:00
Teknium	763b10c320	fix(gateway): run session-finalize plugin hooks off-loop and bounded ... Session-finalize hooks ran synchronously on the gateway event loop from three call sites (shutdown drain, session-expiry watcher, /new reset). A plugin hook doing heavy blocking work froze the whole loop: adapter heartbeats stopped, the drain machinery could not run, and systemd eventually SIGKILLed the process mid-export. Observed live on a multi-day 4.7G session where the nemo_relay observability plugin serialized a full-session ATIF trace inside on_session_finalize. Changes: - gateway/run.py: new GatewayRunner._finalize_session_off_loop() dispatches hermes_cli.lifecycle.finalize_session via the gateway executor under asyncio.wait_for (10s budget), mirroring _cleanup_agent_resources_off_loop (#53175). Shutdown finalize and the session-expiry watcher now use it. - gateway/slash_commands.py: /new reset path uses the same helper. - plugins/observability/nemo_relay: ATIF export is now bounded (HERMES_NEMO_RELAY_ATIF_EXPORT_TIMEOUT_S, default 30s) and skipped entirely for sessions whose Relay scope operations already errored (their exporter state is unreliable and the export can be pathologically slow). - tests/gateway/test_finalize_session_off_loop.py: regression tests proving the loop stays live under a wedged hook and the budget is enforced.	2026-08-15 14:20:27 -07:00
Teknium	26aa12337a	feat: /save exports the current session as json, md, or html on all platforms ... Rework of salvaged PR #6372 (@ag9920) onto current main: - /save promoted from CLI-only JSON snapshot to a cross-platform session export: `/save [json|md|html] [filename] [redact]` on CLI and every gateway platform (sent as a document via adapter.send_document). - Rendering routes through the existing shared renderers (hermes_cli/session_export.py + session_export_html.py) instead of the PR's new hermes_state formatter — new helpers normalize_save_format / render_session_for_save / default_save_filename are shared by both surfaces. - `redact` arg runs the export through the force-mode secret redaction pass (session_export_md.redact_session_data) before writing. - Gateway handler awaits AsyncSessionDB correctly, sanitizes user-supplied filenames with basename, and lands in gateway/slash_commands.py (the handlers moved out of gateway/run.py since the PR was authored). - /export stays profile export (name collision resolved: session export lives on /save). - Slack 50-slash cap curation: /platform moves to the /hermes-only set to free a native slot for /save (parity test updated rationale comment). - Folds in PR #62268 (@briandevans): None title/model coalescing in the single-session HTML export. Closes #4249. Closes #51200.	2026-08-15 02:04:49 -07:00
ag9920	23ad35b7fa	feat: Add /export command for session export (Markdown/JSON) to CLI and Gateway	2026-08-15 02:04:49 -07:00
doncazper	6a375a24a5	fix(gateway): timestamp shared stderr output	2026-08-15 01:04:19 -07:00
Teknium	059e346422	fix(gateway): abort-aware parallel startup connects (restart race parity)	2026-08-14 21:57:41 -07:00
Teknium	c22815fcaa	fix(gateway): cap Telegram cold-start connect so startup reaches running fast (#85993) ... The initial (pre-running) connect awaited during gateway startup now uses a capped 45s budget for Telegram instead of the full 180s (#67498) budget. On timeout the platform is queued for the reconnect watcher, which retries with the full budget and is_reconnect=True (preserving the offline update queue, #46621). Combined with the parallel startup connects, an unreachable Telegram no longer holds the whole gateway out of the running state.	2026-08-14 21:57:41 -07:00
EvanProgramming	d86c67dc7e	fix(gateway): connect messaging platforms in parallel at startup (#83791) ... GatewayRunner.start() previously awaited each platform's connect() (with its own timeout) in a serial for-loop. A single slow/failing platform (e.g. Telegram behind a dead proxy) delayed every later platform's connect by a full timeout window, cascading one platform's failure onto WeChat/QQ/etc. Now the slow connect() calls run concurrently via asyncio.gather while the serial pre-filter (checks, adapter creation, handler wiring) and the single-threaded result aggregation (shared-state mutation, error handling) are unchanged. A failing platform no longer blocks the others. Adds regression tests proving connect() calls overlap and that one failing platform leaves the others connected.	2026-08-14 21:57:41 -07:00