The salvaged fix persisted the recovery note unconditionally — correct for
the synthesized empty auto-resume turn, but a user who typed real text while
resume was pending would get the [System note: ...] scaffold persisted as
their own words, leaking scaffolding into the durable transcript (the same
class as #81841 on the assistant side).
_prepare_resume_pending_message now persists the note only when the original
message is blank; real text persists verbatim while the model still receives
the wrapped note. Whitespace-only counts as blank. Tests cover all three
shapes.
Review follow-up on the salvaged #88965 work:
- test_goal_command_slow_db_init_still_persists: drop the 4s slow-init
loop-gap harness (wall-clock gap assertions on shared CI runners are
their own flake class; loop-freeze bounds are already covered in
test_goals_db_bootstrap_off_loop.py). The persistence contract keeps
its discriminating power by shrinking the monkeypatched init window
(0.2s) under a 0.8s slow init — the window-only path still fails it.
- test_slow_construction_does_not_block_the_loop: monkeypatch both
bootstrap windows down (0.3s/0.05s) and shrink the blocking init from
6s to 1.5s; the two-window contract is what's under test, not the
production constants. Adds an elapsed ordering assertion so the
kick-vs-in-flight window distinction stays pinned.
Combined wall time for the pair: ~10.5s -> ~1.4s.
A fresh state.db init (schema DDL, FTS tables, first config import)
measures ~300ms warm on a fast machine. The gateway constructs
GoalManager on the event-loop thread, and a cold cache ran that init
behind a 0.25s bootstrap grace window: on a slow CI box the /goal set
path's waits expired and save_goal silently no-oped — the reply said
"Goal set (7-turn budget)..." but nothing persisted, and a fresh
GoalManager read back no state (first assertion passes, second fails).
Two changes, one per caller shape:
- Async callers (_get_goal_manager_for_event,
_get_heartbeat_manager_for_event, _post_turn_goal_continuation, and
the heartbeat poller) warm the SessionDB cache off-loop through the
context-preserving executor before constructing the manager (shared
_warm_goals_session_db helper). The loop never blocks and the first
write lands at any init duration. A bare to_thread would lose the
per-turn profile home override under multiplex; the executor hop
keeps it (same pattern as the goal judge path).
- Sync callers (heartbeat persistence, _goal_still_active_for_session)
cannot await, so the bootstrap windows stay: the call that starts the
bootstrap waits a one-time init window (1.5s) instead of the short
per-call window (0.25s), giving healthy cold inits room to land while
a contended migration still degrades to None with only a bounded
one-time stall. The bootstrap thread binds the caller's home as a
contextvar override so a multiplexed worker cannot cache the default
profile's DB under another profile's key.
save_goal and heartbeat save_state now log at WARNING when they drop a
write, because the reply has already told the user the state was set.
Regression test pins the contract: init past the window, write
persists, loop gap under 2s (the flake-policy floor for wall-clock
bounds; the slow-init margin grew to match, so the test still tells
on-loop from off-loop).
Independent diagnosis + measurement by jackulau (#88965 review); the
off-loop warm-up shape follows their harness table. Simplify-code
review (4-agent) contributed the helper extraction and the poller
warm-up.
Third CI hit today for tests/gateway/test_goal_verdict_send.py (twice on
salvage PRs, once on main's own push run), always the same shape:
adapter.sends == [] after the full drain.
Mechanism (reproduced, not log-read): the tests call GoalManager.set() on
the event-loop thread. _get_session_db() refuses to construct SessionDB on
a loop thread (loop-liveness guard from the 2026-08-14 crash-loop fix) and
waits only _DB_BOOTSTRAP_LOOP_WAIT_S=0.25s for the background bootstrap.
On a loaded CI runner the init overruns that window, set() degrades to a
silent no-op by design, the goal never exists, and the continuation path
correctly does nothing — so no amount of drain-waiting helps (the #88975
de-flake addressed a different, downstream race).
Fix: the hermes_home fixture pre-warms the SessionDB cache from its sync
context (direct construction path), so the loop-thread set() always finds
a cached DB. Production behavior untouched.
Proof: injecting a 0.4s-slow SessionDB.__init__ reproduces the exact CI
failure on the old fixture and passes 2/2 with the pre-warm.
test_session_chat_stream_treats_pre_existing_poisoned_row_as_no_model
asserted mock_run.call_args right after the 200 status, but the stream
handler runs _run_agent inside asyncio.create_task(_run_and_signal())
and response.prepare() returns the 200 before that task necessarily
starts — on loaded CI runners call_args was still None (TypeError:
cannot unpack non-iterable NoneType). Draining the SSE body (resp.text())
joins the stream end, which guarantees the runner task completed.
test_goal_verdict_send used fixed asyncio.sleep(0.05) waits before
asserting on sends/enqueues produced by spawned tasks; replaced with a
bounded _drain_until() poll (5s cap, returns as soon as the condition
holds) so the asserts stay exact without the fixed-delay race.
These three tests red-flagged unrelated main pushes and PR runs on
Aug 18 (runs 32099139396, 32101135100, 32106224479, 32101070064).
Three cases in TestCreateAgentModelRecovery: the alias is never
cached, a poisoned cache is never served, and legitimate
last-known-good recovery still works.
Salvaged from #72739 (net diff onto current main; the read-side legacy-row
guard now flows through the shared _stored_session_model() helper that
landed in #88751, keeping one resolver for both chat sites).
POST /api/sessions persisted the advertised virtual alias (hermes-agent)
whenever the request omitted model or echoed the alias back; later turns
replayed it upstream as a real model id and every turn on the session
failed. Null the alias in _session_runtime_request_from_body() — shared by
session create, chat, chat-stream, and model-lock — and stop the
create-handler's raw-body fallback from bypassing that normalization.
Two bugs found by a REAL two-gateway live test (two isolated HERMES_HOMEs,
bravo running the api_server platform, alpha's agent autonomously running
`hermes peer dm` from its Bot Chat protocol; reply relayed correctly and
persisted in bravo's canonical Bot Chat):
1. peer dm parsed the session-create response flat, but api_server wraps
the row: {"object": "hermes.session", "session": {...}} — every first DM
to a fresh peer failed with "Peer did not return a session id" (and the
orphaned Bot Chat then 400'd retries with duplicate-title). Parse the
wrapped shape; the test fake now mirrors the real response shape so this
class can't pass green again.
2. api_server: a session created with no model persists the advertised
virtual model ("hermes-agent") on the row; session chat then replayed it
as a REAL model id and the provider 400'd ("hermes-agent is not a valid
model ID"). _request_agent_overrides already filters the virtual model
for per-request bodies — apply the same filter to the stored session
model at both chat sites (sync + stream), so it means "gateway default"
exactly like the request-body path.
Live E2E transcript (bravo's Bot Chat, via /api/sessions/{id}/messages):
user: Message from 🤖 alpha (@alpha): What is your callsign?
assistant: CALLSIGN-BRAVO-7
peer cmd unit suite 10/10 with the corrected fake.
Follow-up to the salvaged #88632 (Jack Lau) fix for #88532, extending the
same repair to the sibling frozen-at-init handle and closing the handle
lifecycle gap the per-path cache introduces.
1. GatewayRunner._session_db had the identical bug class: bound once as
AsyncSessionDB(SessionDB()) in __init__ on the root home, while /resume,
/title, /history and session search all execute inside
_profile_runtime_scope on a multiplexed gateway. Convert it to the same
property-with-pin pattern: per-access resolution of _default_db_path(),
one cached AsyncSessionDB per resolved path under a lock, and assignment
preserved as an explicit pin (many suites install fakes or None).
Construction-time priming keeps the #88235 init-failure broadcast at
startup.
2. Handle lifecycle: the per-path caches accumulate one open SessionDB per
profile served, but the shutdown path closed only store._db /
runner._session_db - which now resolve just the shutdown task's own
(root) scope. Secondary profiles' handles would strand their WAL write
locks until process exit, recreating the abandoned-handle leak
b454e4da76 fixed and breaking --replace restarts with 'database is
locked'. Add close_all_db_handles() / close_all_session_db_handles()
sweeps and call both from the gateway teardown path. SessionDB.close()
is idempotent, so the root handle being closed by both the legacy loop
and the sweep is safe.
Tests: sweep coverage plus a runner-property scope/pin/cache test in
tests/gateway/test_multiplex_session_db_profile_scope.py.
Fixes#88532.
A multiplexed gateway serves every profile from one process, but
SessionStore bound a single SessionDB during __init__:
self._db = SessionDB()
SessionDB(db_path=None) resolves _default_db_path() at call time and
does follow the context-local HERMES_HOME override, so the path
machinery was already correct. The problem was when it ran: at
construction, on the process's own root home, long before any inbound
event enters _profile_runtime_scope. Every profile's rows therefore
landed in the root state.db, even though the scope had redirected
get_hermes_home() correctly for the turn (that helper's own docstring
lists "sessions" among what it scopes).
The rows still carry the right profile_name, stamped from
source.profile by the same handler, so nothing in the data looks wrong.
The only visible symptom is the desktop listing a profile's session
under the default bot: _open_session_db_for_profile opens
profiles/<name>/state.db, which never received the write.
Look the handle up through a property instead, resolving the active
scope per access and caching one handle per resolved path so a hot
inbound path opens SQLite once per profile rather than once per
message. Construction stays under the cache lock so a concurrent first
message on a profile cannot open and then leak a second handle.
Assignment is preserved as an explicit pin, which is what the existing
suites rely on when they install a fake handle or disable the DB with
store._db = None, and a pin keeps winning across scope changes.
Behavior is unchanged when no profile scope is active, so single-profile
gateways resolve exactly the path they did before. This does not migrate
rows that already landed in the root store; those stay where they are.
Healthy IPv4-first connect is the new default path, so two transports
were warning on every successful initialize. Keep warning only when a
literal actually failed first. Also restates the transport docstring
and docs to match IPv4-first, hostname last.
A blackholed IPv6 path to api.telegram.org never errors, so
_await_with_thread_deadline never fires and connect hangs at
"attempt 1/8". Known A-record IPs connect over IPv4 immediately.
DoH timeout now fail-opens to the seed IPv4 list instead of the
hostname. Hostname stays last for IPv6-only hosts.
Closes#87015
Matrix m.audio/m.file/m.video events populate content.body with the
uploaded filename when the sender adds no caption. The adapter already
blanks that for m.image (PR #16821, issue #13482) but not for audio,
file, or video msgtypes, so the filename survives into event.text and
is appended after the transcript, where the model reads it as the
user message rather than as transport noise.
Extend the existing adapter-level blanking: add
_looks_like_matrix_media_filename() with the same conservative
heuristic (single token, no whitespace, no path separators, known
media suffix or mimetypes audio/video match) and apply it at the
media message handler for m.audio, m.file, and m.video.
Salvage of #87968 by @AiwendilInTheWoods — reworked from shared
gateway path to adapter-level fix for consistency with the existing
m.image blanking.
Drives the real imported-before-installed transition: seeds a fake SDK
into sys.modules, resets the adapter globals to the placeholder the
fallback block leaves behind, stubs the installer to a no-op, then calls
check_telegram_requirements() and asserts no placeholder survived.
Asserting over the whole stubbed set rather than one name means the next
symbol dropped from the rebind list fails here too, naming itself:
AssertionError: lazy install left these bound to the import-time
placeholder: ['TypeHandler']
Originally written by Tair Asim in #85607, which was closed as a
duplicate of this PR. Moved into its own file to match the convention
already used by test_discord_lazy_install_views.py and
test_feishu_lazy_import.py, and widened from three hard-coded symbol
checks to the full set.
Co-authored-by: Pavel Lesyuk <paul.lesyuk@gmail.com>
The freshness window (updated_at >= live process create_time - 2s) had a
P1 boundary hole: a stale failure written by the PREVIOUS process
immediately before a fast restart landed inside the slack and was
aggregated; if that platform was then removed, the new process never
replaces the entry and NAS stays degraded indefinitely.
Replace clock heuristics with persisted writer identity:
- write_runtime_status now stamps every platform entry with the writing
process's (writer_pid, writer_start_time) — the same PID-reuse
fingerprint the liveness checks use, so a recycled PID never
masquerades as the original writer.
- The aggregation ownership filter requires exact equality between an
entry's stamp and the profile's validated live gateway process
(get_runtime_status_running_pid + _get_process_start_time). No slack,
no timestamps. Legacy entries without a stamp fail closed.
- Writer stamps are process recon (same class as the auth-gated
gateway_pid) and are stripped from all /api/status projections, both
active-profile and merged cross-profile entries.
Near-boundary regression test: prior-process entry stamped 100ms before
restart is excluded; recycled-pid-different-fingerprint excluded;
legacy no-stamp excluded; current-process entry kept.
Scoped credential locks (Telegram bot token, Discord bot token, etc.) are
machine-global, but the conflict error only reported the holder's PID:
Telegram bot token already in use (PID 559). Stop the other gateway first.
On multi-profile hosts (e.g. hosted instances running 13 profiles), a bare
PID gives the operator no way to tell WHICH profile owns the credential —
the exact failure mode observed on zerocool-9781, where the 'default'
profile was misconfigured with the same bot token as 'lead-gen-outreach'
and logged an unattributable conflict every ~5 minutes (4,602 rows).
Fix:
- acquire_scoped_lock() now stamps a 'profile' label on lock records,
inferred from the process HERMES_HOME (<root>/profiles/<name> layouts,
'default' for the root home). Omitted when not inferable.
- New scoped_lock_owner_label() resolves the owning profile from a lock
record: prefers the explicit field, falls back to inferring from the
persisted hermes_home for locks written before the field existed.
Labels are validated against the profile-id grammar before use (lock
files are plain JSON on disk and the label flows into log lines and a
suggested CLI command).
- _acquire_platform_lock() conflict message now names the owning profile
and gives the correct remedy:
Telegram bot token already in use by the 'lead-gen-outreach' profile
gateway (PID 559). Stop that gateway first
(hermes --profile lead-gen-outreach gateway stop).
Records with no attribution signal keep the original PID-only wording.
Testing:
- New TestScopedLockOwnerLabel suite covering label inference (named,
Docker, root/default, unknown layouts), grammar validation, explicit-
field preference, hermes_home fallback, and legacy/malformed records.
- acquire_scoped_lock tests for profile stamping and omission.
- Adapter-level tests for profile-attributed, legacy-home-inferred, and
PID-only conflict messages.
- 76/76 targeted gateway tests pass; broad gateway suite failures are
baseline-identical (verified via git stash comparison). Ruff clean.
MEDIA_TAG_CLEANUP_RE (and MEDIA_EXTENSIONLESS_TAG_RE) only recognized
ASCII terminators after a MEDIA:<path> tag. Chinese-language agent
output naturally writes MEDIA:D:\...\zhibao.pdf（782.6 KB）or ...pdf：内容 —
the full-width punctuation failed the trailing lookahead and the
attachment was silently dropped (cron even reported 'delivered') (#88038).
Both lookaheads now accept a CJK full-width terminator set (（）〈〉《》：，。；
！？、curly quotes【】) alongside the ASCII set. The #68773 adjacent-tag
splitting guard is covered by a regression test.
Streamed replies return None so the adapter does not send twice.
The /loop hook then saw empty text and never ran, so
awaiting_response stayed true and later ticks never fired.
Stash the delivered text on the event and use it for the post-turn
hooks. /goal uses the same path.
Tests: tests/gateway/test_loop_command.py
With gateway.multiplex_profiles enabled, the primary Telegram message
handler is the closure returned by _make_default_profile_message_handler(),
so its __self__ is absent. The early intake filter
(_is_user_authorized_from_message) recovered the GatewayRunner via
self._message_handler.__self__ and, finding none, fell back to env-only
authorization — never evaluating the configured chat allowlist through
GatewayRunner._is_user_authorized(). Every non-global sender was then
default-denied in an explicitly allowlisted group.
Prefer the platform-bound authorization callback registered via
set_authorization_check(): it routes through the runner's full auth chain
(platform + group allowlists, pairing store, allow-all) and survives the
closure wrapping, whereas the bound-handler lookup does not. The bound
handler remains the fallback for setups without a registered callback, and
the pairing-passthrough guard for unknown DMs is preserved.
Fixes#87132
The fire-and-forget handler only caught asyncio.CancelledError and
Exception, so a SystemExit/KeyboardInterrupt escaping a turn (e.g. a
plugin calling sys.exit() in a tool call or summary-LLM path) skipped
the user-facing failure notification and surfaced only as 'Task
exception was never retrieved' — radio silence for the user.
Catch BaseException instead; send the failure notification first, then
re-raise SystemExit/KeyboardInterrupt to preserve shutdown semantics
for the loop's own signal handling. Other BaseExceptions stay contained.
Closes#86651
Session-finalize hooks ran synchronously on the gateway event loop from
three call sites (shutdown drain, session-expiry watcher, /new reset).
A plugin hook doing heavy blocking work froze the whole loop: adapter
heartbeats stopped, the drain machinery could not run, and systemd
eventually SIGKILLed the process mid-export. Observed live on a
multi-day 4.7G session where the nemo_relay observability plugin
serialized a full-session ATIF trace inside on_session_finalize.
Changes:
- gateway/run.py: new GatewayRunner._finalize_session_off_loop()
dispatches hermes_cli.lifecycle.finalize_session via the gateway
executor under asyncio.wait_for (10s budget), mirroring
_cleanup_agent_resources_off_loop (#53175). Shutdown finalize and
the session-expiry watcher now use it.
- gateway/slash_commands.py: /new reset path uses the same helper.
- plugins/observability/nemo_relay: ATIF export is now bounded
(HERMES_NEMO_RELAY_ATIF_EXPORT_TIMEOUT_S, default 30s) and skipped
entirely for sessions whose Relay scope operations already errored
(their exporter state is unreliable and the export can be
pathologically slow).
- tests/gateway/test_finalize_session_off_loop.py: regression tests
proving the loop stays live under a wedged hook and the budget is
enforced.
The initial (pre-running) connect awaited during gateway startup now uses
a capped 45s budget for Telegram instead of the full 180s (#67498) budget.
On timeout the platform is queued for the reconnect watcher, which retries
with the full budget and is_reconnect=True (preserving the offline update
queue, #46621). Combined with the parallel startup connects, an unreachable
Telegram no longer holds the whole gateway out of the running state.
The previous concurrency assertion (slow_start < fast_end) was true under
BOTH the serial and parallel implementations, so it proved nothing -- it
even passed against the old serial code on main. The only assertion that
distinguishes the two is that the fast platform finishes before the slow
one (fast_end before slow_end), which is only possible when the connects
overlap.
Switch the test to record connect start/end events in arrival order
(clock-resolution independent) and assert fast_end precedes slow_end. This
also fixes the Windows failure @zuowen7 reported: time.monotonic() has only
~15 ms resolution there, so two parallel connects could land on the same
tick and defeat any wall-clock comparison -- event ordering cannot.
Verified the new test fails against origin/main (serial) and passes against
this branch (parallel).
GatewayRunner.start() previously awaited each platform's connect() (with its
own timeout) in a serial for-loop. A single slow/failing platform (e.g.
Telegram behind a dead proxy) delayed every later platform's connect by a full
timeout window, cascading one platform's failure onto WeChat/QQ/etc.
Now the slow connect() calls run concurrently via asyncio.gather while the
serial pre-filter (checks, adapter creation, handler wiring) and the
single-threaded result aggregation (shared-state mutation, error handling)
are unchanged. A failing platform no longer blocks the others.
Adds regression tests proving connect() calls overlap and that one failing
platform leaves the others connected.
mark_running_jobs_interrupted skipped legacy fires without a registered
durable owner entirely — correct for the persisted last_status write
(no owner fence to protect a replacement run), but the gateway shutdown
path also uses the returned ID list to deliver interrupted-cron notices
while adapters are still connected (#82232). Keep the persistence skip,
but include the job in the returned list so the user is still told.
When the shutdown drain times out and kills an in-flight cron job, the
job's owner is never told. The cron worker does try: `_is_interrupted()`
forces the failure path with an honest "interrupted by gateway shutdown"
error, and failed jobs always deliver. But that worker is a thread, it
reaches `_deliver_result()` asynchronously, and by then
`_bounded_adapter_teardown()` has closed the transport. The reporter of
Worse, the loss is silent twice over: `_consume_interrupted_flag()`
returns True — the gateway already wrote `last_status` — so
`mark_job_run()` is skipped, and the `delivery_error` from the failed
send is discarded with it. The run's only trace is a generic line in
jobs.json.
The gateway already owns the right window. `_notify_active_sessions_of_
shutdown()` runs while adapters are up, precisely so shutdown messages
can be sent — but it iterates `_running_agents`, and cron work lives on
the scheduler's own thread pool. Same structural blindness already fixed
for counting (#60432) and draining (#63529), never fixed for notifying.
So notify from the post-interrupt phase, which is the last point where
the transport is still up: `_kill_tool_subprocesses()` now returns the
job IDs it marked, and `_notify_interrupted_cron_jobs()` sends each one's
owner a notice on the job's own resolved delivery targets. Adapter
teardown order is untouched — it is load-bearing for #53175 and #8202.
Jobs with `deliver: local`, and `deliver: origin` jobs with no resolvable
origin (#43014), resolve to zero targets and stay silent. Per-platform
`gateway_restart_notification: false` is honoured, matching the chat
path. Every failure is swallowed so a wedged adapter cannot extend
shutdown.
Second, when the interrupted flag short-circuits `mark_job_run()`, the
delivery failure is now persisted on its own via `update_job()`, so a
notice that still cannot be sent is at least recorded. `update_job()`
rather than a second `mark_job_run()`: the latter also advances
`next_run_at` and the repeat counter, and running that twice for one run
would skip a fire or auto-delete the job early.
Fixes#82232. Related: #82161, #82224.
CI slice 5/12 caught two ways the new cron budget broke `_stop_impl_body`
for callers that are not real GatewayRunner instances:
- `_FakeGateway` in test_shutdown_cache_cleanup.py borrows `_stop_impl`
without subclassing, so it never picked up the class-level
`_cron_drain_timeout` default and raised AttributeError. Read it through
the getattr-guard convention the same function already uses for its
liveness-guard machinery.
- The same double overrides `_drain_active_agents(self, timeout)`, so
passing the cron budget raised "takes 2 positional arguments but 3 were
given". The double now mirrors the real optional parameter. It is the
only override in the tree; test_startup_restart_race.py uses AsyncMock,
which accepts any signature.
Verified against a stashed clean tree: the 22 gateway test files that
still fail locally fail identically with and without this branch (80 = 80,
empty set difference both ways) — they are pre-existing Windows-only
failures (setsid, POSIX modes) unrelated to this change.
`agent.restart_drain_timeout` defaults to 0 and governed every class of
in-flight work at once. That default is deliberate for chat turns: the
gateway announces the restart to the user and pre-marks the session
resume_pending, so interrupting one is cheap and recoverable.
A cron run has neither property. Nobody is waiting on it, it is written
to jobs.json as a permanent failure, and a recurring job simply skips to
its next schedule. Sharing the chat budget meant `_drain_active_agents()`
short-circuited on `timeout <= 0` before entering the wait loop, so the
drain reported `drain took 0.00s, timed_out=True, cron_at_start=1,
cron_now=1` — it detected the job and killed it anyway.
Cron work now drains on its own deadline, `agent.cron_drain_timeout`
(default 30s, 0 opts out). The floor is clamped to the shutdown-watchdog
leash minus a teardown reserve, so the longer wait can never consume the
post-drain cleanup window: being SIGKILLed mid-cleanup would leave the
job wedged at `last_status=running`, strictly worse than the bug. Being
bounded also means a cron-triggered restart cannot deadlock on itself.
The `timeout <= 0` special case is gone — an expired deadline expresses
the legacy "interrupt immediately" behaviour, so `timed_out` is always
computed from real state instead of asserted up front. The drain-timeout
warning now reports the elapsed wait rather than the configured budget,
which is what made "timed out after 0.0s" so confusing in the report.
Chat-only shutdowns are unchanged: `restart_drain_timeout: 0` still
interrupts chat turns immediately.
Relates to #82161 (complements #82195, which removes the `hermes update`
self-deadlock that triggered the reported instance).
Teknium