A gateway process opened state.db from ~12 call sites, each minting its
own writer connection, self._lock, close-time WAL checkpoint, and
token-writer thread. With N independent writers on one WAL file, one
connection's close-time checkpoint could race another's growth — the
lost/reordered-page-write signature across 11+ incidents (#90837).
Adds hermes_state_registry.py: a process-wide, per-path, refcounted
shared registry owning the writer boundary.
- acquire(path): same resolved path returns the same instance (one
writer connection, one lock, one token-writer thread) for every
long-lived in-process caller (gateway runner, SessionStore, per-agent
lazy recall, cron per-job, mirror, channel_directory, slash_commands,
shutdown_flush, session_search, react_to_message, delegate, mcp_serve,
auto_archive, tui_gateway).
- close() on a shared instance is a NO-OP — the registry owns the
lifecycle, so one caller's close can never tear down a writer other
callers still hold.
- Generation-aware retirement on inode change: a replaced state.db
RETIRES the live generation (never lent again) but keeps it alive for
existing holders; release is object-keyed so holders of the old
generation drain it independently of the new one. The old
generation's own write path still fails with the typed
StateDbReplacedError (existing protection, unchanged).
- Replacement-open failure leaves NO registry entry for the path —
the next acquire retries fresh, never hands out a closed stale object.
- All teardown runs OUTSIDE the registry lock: a final release's WAL
checkpoint can never stall acquisition for every state.db.
- close_shared_session_dbs() at gateway shutdown drains every
generation (live + retired) as the final safety net.
CLI one-shots, recovery flows, and read-only cross-profile opens keep
using SessionDB() directly with their own close() — only long-lived
in-process sites route through the registry.
References #90837 (root-cause tracker stays open: the #10 EOF signature
and the WAL-lifecycle A/B verdict remain under investigation there).
The "Connecting to Telegram (attempt N/8)…" line logs at WARNING and
reaches the gateway's default stderr handler, but the matching
"Connected to Telegram (… mode)" line was INFO and went to the log file
only. A healthy startup therefore looked permanently hung at
"attempt 1/8" on the terminal — the logging-illusion half of #90835.
Promote the success line to WARNING so both sides of the connect
transition share the same console sink; a genuine hang is now the
absence of the success line. Adds an AST-level regression test pinning
the level pairing. Sibling adapters (homeassistant, wecom) log both
sides at INFO, so they don't have this asymmetry.
Fixes#90835
@teknium1's maintainer-side review found no blocking defect on 09004753c9 and
listed five cleanups. All five are here.
1. scratch/repro_96811.py is deleted. It would have landed on main as a
tracked file: scratch/ is not gitignored and has never existed on main, so
this PR was creating the directory. Nothing referenced the probe, and
TestConversationGenerationRotates / TestGenerationSurvivesPruning /
TestPeerIdentityIsSourceQualified already carry all four of its stages, so
it is dropped rather than parked under tests/.
2. Upgrade notes are written into this commit body (below) and the PR body.
There is no committed changelog to add them to: scripts/release.py
generates .release_notes.md from commit SUBJECTS at release time, and
.gitignore keeps that file out of the tree.
3. declared_conversation_scope() now reads the sessions row ONCE. The fork
verdict and the source the peer queries match on both live on that row, and
asking for them separately read it twice per resolution. The new
SessionDB.declared_scope_identity() returns the pair and keeps the marker
rules beside is_explicit_fork_child() instead of re-implementing them in the
caller. A SessionDB that does not expose the combined view keeps the
original two-call path, so nothing that predates it changes behaviour --
including the three doubles that certify the fail-closed contract, which are
untouched. TestOneIdentityReadPerResolution pins the single read, the
two-call fallback, the fail-closed degrade and the fork refusal; removing
the fold turns the first of those red.
The third read stays: the generation lives in conversation_generations, a
different table, and cannot be folded into a sessions lookup.
4. _declared_conversation_session() documents the concurrent first-turn race.
Two simultaneous first requests on one declared key can each miss the
lookup, mint a row and both bind, because each row is unkeyed at bind time
and the mismatch guard does not fire. That converges rather than crossing:
both rows carry the same key under the same source, so the lookup returns
the later one for every subsequent reply and the earlier row is an abandoned
transcript, never another conversation's identity.
The same docstring still claimed the generation was durable in
sessions.end_reason and that "nothing here needs a counter". That stopped
being true in 09004753c9, which moved the generation into
conversation_generations precisely because deriving it from prunable session
rows was ABA. Corrected, along with the same stale sentence on
TestConversationBoundariesRotate.
5. conversation_generations rows are now documented as deliberately never
collected, rather than merely uncollected. Dropping one resets that peer to
"no generation", so its next boundary writes 1 again and re-issues a gwk_
scope a retired conversation already used -- the exact ABA the table exists
to close. Worth stating because the repo already carries both patterns a
maintainer would extend: delete_session() cascades to messages, and
gateway_hygiene_state is already swept by session_key.
Upgrade notes, one-time on merge:
- One cold prompt-cache bucket per keyed conversation. Every gateway platform
declares gateway_session_key, so each keyed conversation's affinity scope
moves once from its compression-lineage root session id to the gwk_ hash.
One cache miss per live conversation, on its next turn only.
- hermes status counts more sessions. A declared API conversation is now
recorded as a keyed row and appears in "Active: N session(s)" where it was
invisible. Those sessions already existed; only their visibility changes.
- A database upgraded mid-conversation starts with no generation and takes its
first from the next boundary written, so a conversation that reset before the
upgrade shares its predecessor's scope once. One warm bucket, never a crossed
identity.
Verified on this head: 55 in test_declared_conversation_scope.py (51 + 4 new),
33 in test_prompt_cache_scope.py, 49 in test_api_server_declared_conversation.py,
25 in test_api_server_runs.py, 109 in test_api_server.py, 12 in
test_cross_process_turn_lease.py, and 526 across test_hermes_state.py +
tests/hermes_state/ + tests/state/. ruff clean.
Found in review by @teknium1.
Refs #96811
Four blockers from @andrexibiza's reviews of 28a2d7f0ee and dc7865765c. The
first two are defects I introduced in 99f2d4394f by replacing the wrong
occurrence of an identical call site.
1. _run_agent raised NameError on every opted-in declared bind. Its worker
finally evaluated `if _declared_selected:`, a local of _handle_responses /
_handle_runs that is neither a parameter nor an enclosing binding here, so
the successful declared-key paths failed at settlement after the agent run.
bind_declared_conversation already IS the gate; the inner name is gone.
2. /v1/runs never received the gate at all -- it landed on _run_agent instead.
_run_sync bound unconditionally, so an explicit body session_id that existed
with an empty session_key was adopted by the header key even though the
header lost precedence. It now carries the same gate.
3. COUNT(*) + MAX(ended_at) over session rows cannot prove non-reuse.
delete_session() deletes the selected row and bulk prune selects ended rows,
so the aggregate can return a pair it already emitted:
(1,T1) -> (2,T2) -> delete boundary B -> (1,T1), handing a new conversation
a retired affinity identity. The backwards-clock shape needs no pruning at
all. The generation now lives in a conversation_generations table keyed by
(source, session_key), advanced by _bump_conversation_generation inside the
same transaction that writes each boundary -- outside prunable session
history, wall-clock-free, and increment-only. end_session() and
promote_to_session_reset() both advance it, and only when they actually
wrote a boundary, so a repeated end cannot double-count.
4. The carrier could be memoized under the wrong source. _agent_source() fell
back to agent.platform before the row landed while persistence uses
_session_source_for_agent(), which honors HERMES_SESSION_SOURCE. Because a
declared scope is non-None immediately, resolve_prompt_cache_scope memoizes
it and never re-resolves once the authoritative row appears, so under an
override both sides of a /new read the platform domain and hashed the same
scope. The pre-row path now uses the persistence resolver itself.
Coverage answers the review's specific objection that mocked tests proved the
mock rather than the path. TestRealRunAgentSettlement stubs _create_agent and
lets the real _run_agent settle; the /v1/runs case persists an unkeyed explicit
row and waits for the worker to retire before asserting. Both were verified by
mutation: reinstating the inner name fails two of them, and removing the
/v1/runs gate fails the explicit-session one. The first version of that test
passed with the gate removed -- it asserted before settlement -- and would have
been the same empty proof the review called out.
TestGenerationSurvivesPruning covers deleting the newest boundary, deleting
every boundary, the backwards-clock-then-prune shape, compression and
accidental ends not advancing it, repeated ends not double-counting, promotion
advancing it, unkeyed rows advancing nothing, and peer scoping.
TestSourceOverrideDomain covers the override across a reset.
Found in review by @andrexibiza, whose analysis located each of these
defects and specified what a correct fix had to prove.
Refs #96811
Co-Authored-By: Andrex Ibiza, MBA <andrexibiza@gmail.com>
The review asked for "real handler + DB coverage for both precedence paths".
The first pass did not deliver that: TestBindFollowsPrecedence restated the
gate expression inside the test, so it asserted a copy of the rule rather than
the rule, and would have stayed green if the handlers stopped applying it.
These drive POST /v1/responses and POST /v1/runs over real routes with a real
adapter and a real SessionDB:
- the declared key selects and records the conversation, and the row it
produces carries the key;
- three replies on one declared key land on one session id;
- an undeclared request keeps a per-request id and records nothing;
- a request carrying conversation A's previous_response_id plus a foreign
header key settles on A, records nothing, leaves A's own key intact, and the
foreign key still cannot recover A -- the end-to-end shape of the blocker;
- /v1/runs, which owns its agent lifecycle rather than routing through
_run_agent, settles on the declared conversation, and an explicit body
session_id outranks the header key without rebinding it.
The _run_agent stand-in creates the session row the way
AIAgent._ensure_db_session does and performs the bind the way _run_agent's
finally block does, so the assertions land on real rows instead of on a mock's
call args. /v1/runs is captured at _create_agent for the same reason.
The restated-gate tests are kept as the cheap unit layer beneath these.
Found in review by @andrexibiza, whose analysis located each of these
defects and specified what a correct fix had to prove.
Co-Authored-By: Andrex Ibiza, MBA <andrexibiza@gmail.com>
Both blockers from @andrexibiza's review of 28a2d7f0ee.
1. The generation lookup was not in the same identity domain as recovery.
latest_conversation_boundary() selected on session_key alone, while
_declared_conversation_session() is qualified by (source, session_key).
X-Hermes-Session-Key accepts any authenticated caller-supplied string, so an
API conversation may legally carry the same key as a Telegram row in one
database -- a /new over there rotated this conversation's gwk_ generation
while recovery correctly refused to cross the same line, moving the affinity
identity out from under a physical identity that had not moved.
The boundary read now takes (session_key, source), and the carrier is
'source|key|generation' rather than 'key|generation' -- keying on the string
alone would also collapse two same-key conversations from different sources
onto one routing key, since this value leaves the process verbatim as
OpenRouter's sticky session_id and xAI's x-grok-conv-id. The source comes
from the agent's own session row, falling back to the platform the row will
be created with before it lands.
2. The declared key's stated lower precedence did not survive settlement. Both
handlers let stored_session_id / an explicit body session_id win, then called
_bind_declared_conversation() unconditionally. record_gateway_session_peer()
does SET session_key = ? across compression ancestors, so a request carrying
conversation A's chain plus header key B silently rebound A to B: A could no
longer be recovered by its own key, and B recovered A's session.
Recording is now gated on the declared key having actually selected or
minted the session, on both paths. Behind that gate the bind itself refuses
to overwrite a row already bound to a different key, so a future caller
cannot reintroduce the same defect by opting in wrongly.
test_declaration_outranks_the_lineage_root asserted the pre-qualification
contract by comparing a DB-backed agent against a DB-less one; it now makes the
stronger statement it was written for -- one declared conversation reached
through two different physical ids on the same peer.
Refs #96811
Found in review by @andrexibiza, whose analysis located each of these
defects and specified what a correct fix had to prove.
Co-Authored-By: Andrex Ibiza, MBA <andrexibiza@gmail.com>
POST /v1/responses and POST /v1/runs parse and authenticate the client's
X-Hermes-Session-Key, pass it downstream for memory scoping, and then mint a
throwaway physical session id anyway whenever the client manages its own
history (no previous_response_id chain to carry one forward).
Every conversation-affinity hint Hermes sends is derived from that physical
id, so all four re-keyed on every single reply: prompt_cache_key on both
OpenAI-wire transports, the OpenRouter and Nous sticky session_id, and xAI's
x-grok-conv-id. The conversation never landed back on a warm prefix.
Fix the identity rather than the four consumers. The declared key resolves to
its live session through find_latest_gateway_session_for_peer -- the same
reset-fenced recovery every native gateway platform already uses -- and the
turn records the row it ended on through record_gateway_session_peer, which
AIAgent._ensure_db_session never did (it knows the key and writes the row
unkeyed, so the mapping the next reply needs did not exist).
Because the lookup is fenced on sessions.end_reason, the generation that must
rotate is already durable: session_reset (/new), session_switch, idle, daily,
suspended and resume_pending_expired all return None, so a new conversation
gets a new id and a cold affinity scope, and a retired generation can never be
resolved again. No counter, no new persisted field, and no new precedence rule
in the cache-scope resolver -- /branch, delegate and tool children keep the
isolation of #79161/#79017 byte for byte.
Precedence is unchanged where it already worked: an explicit body session_id
and the previous_response_id chain both still outrank the declared key, and a
request that declares nothing keeps its per-request id. Recording is opt-in
(bind_declared_conversation), so no other _run_agent caller's rows change.
Refs #96811
(cherry picked from commit e7c83dddf36784d1012bf483240ebc7f6b2ef9aa)
The #66887 fix pins the routing index to HERMES_HOME state.db; the
fast-path harness still read entries through the ambient store. Point
get_hermes_home at the test tmp so both are the same file, matching
the new single-store contract.
Second half of #66887. _entries is a single flat dict holding every
profile's keys, so the index it persists to has to be a single file — but it
was read and written through _db, which resolves whichever profile scope is
active. A whole-index rewrite during one profile's turn copied every other
profile's routing rows into that profile's store, and startup, which runs
unscoped, then loaded a different copy than the last writer produced.
That is why the startup recovery pass never sees a secondary profile's crash
marker, which is the half this issue's title names. mark_turn_active()
persists through the single-entry fast path (state.db only, no sessions.json
mirror), so a marker written during a profile's turn landed in that
profile's store and _recover_unclean_sessions(), running with no scope, read
a store that had never heard of it. The turn was silently never promoted to
resume_pending.
Capture the gateway's own home at construction — the store is built at
startup before any profile scope exists — and route the index through it:
_ensure_loaded_locked, _reconcile_recovered_routing_locked,
_persist_routing_data and _save_entry now use _routing_db. A pinned handle
still wins, so suites that install a fake or disable the DB are unaffected.
_prune_stale_sessions_locked is the mixed case and is split accordingly: it
now asks _db_for_key(key) whether each session ended, because that is a
per-session question, while the index write stays on the single store. One
ambient handle previously answered it for every profile at once, which could
prune a live secondary-profile route on the strength of the root store's
copy of that session.
Regression as requested on the issue: mark a turn active under a secondary
profile's scope, then build a fresh store with no scope and run
recover_interrupted_turns(). It promotes exactly one turn to resume_pending
here and promotes zero against the previous behaviour.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review P1 #2. _append_to_transcript_serialized() writes the compression
continuation to child_id BEFORE publishing either _transcript_reroutes or
the _entries update — that ordering is load-bearing for backlog order, so it
must not move. At that moment nothing in the routing index points at the
child, so _db_for_session_id(child_id) missed its scan and fell through to
_db_for_key(None), i.e. the ambient store. The fail-closed guard did not fire
because root is a live handle.
The row therefore targeted root rather than the already-proven parent owner.
With no child row there the append is rejected by the FOREIGN KEY constraint,
the pending queue never drains and the reroute cannot advance; against a
split-brain root the message would instead be written cross-profile.
Record ownership before the mutation instead of moving the publication: a
private _session_owner_hints map carries session_id -> owning key for ids
whose owner is proven but not yet published, consulted by the new
_owner_key_for_session_id() after the index scan misses, and dropped as soon
as routing publishes. Signatures are unchanged, so the existing suites that
stub _append_transcript_message keep working untouched; the map is read
through getattr for stores built via object.__new__.
The regression is physical rather than mocked: an ended compression parent
and a live child that exist only in profiles/fitness/state.db, no active
profile scope, append to the parent, then assert all four effects — the row
lands on the child in the profile store, the pending queue drains, the
reroute and the routing entry advance, and root state.db stays untouched.
Without the hint it fails exactly as the review predicted, on
"FOREIGN KEY constraint failed" against root.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review P1. _profile_home_for_key() returned the same None for three
different states — multiplexing off / legacy agent:main namespace, a named
profile whose directory does not exist yet, and a resolution error — and
_db_for_key() collapsed all of them to the ambient store.
That recreated the very split this change removes. The enrollment bridge
provisions profiles/<name>/ at runtime, so a key such as
agent:fitness:telegram:dm:1 can legitimately be seen first: the first lookup
landed in root state.db, and the next one, after provisioning, in
profiles/fitness/state.db. One qualified session identity, two physical
stores. The resolver-exception path fell open the same way.
Ownership is now tri-state:
- no named owner -> ambient DB (single-profile behavior intact)
- named owner + home -> that profile's DB
- named owner, unresolvable -> None, and a warning; never root
Callers already treat a missing DB as "skip the mutation", which is the
defer-don't-misroute behavior wanted here. _append_transcript_message is the
one path reached with an id the entry-point guard did not check (the
compression-child id), so it now raises explicitly and lets the caller's
retry queue hold the row instead of relying on an AttributeError.
Tests exercise the effect boundary rather than cache state: a named key
before its profile exists leaves root untouched and lands only in the
profile store once provisioned, and a resolver exception fails closed too.
Both fail against the previous two-state behavior by returning a live
SessionDB where None is required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review feedback on the memo introduced with _profile_home_for_key. The
problem is sharper than "no invalidation on profile deletion": caching the
miss pinned a profile that appears AFTER the gateway started to the ambient
store for the life of the process, which is the exact failure this helper
exists to prevent.
That is not hypothetical — an enrollment bridge can provision
profiles/<name>/ at runtime, so a key is legitimately seen before its
directory exists.
Memoize hits only. A miss costs one profile_exists() stat and recurs only
for profiles that genuinely do not exist, so the hot path for real profiles
is still a dict hit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#88734 made SessionStore._db follow the ambient HERMES_HOME so a multiplexed
profile's rows reach its own state.db. That is correct for the inbound message
path, which installs the scope via _profile_runtime_scope. Nothing else does.
_session_expiry_watcher (gateway/run.py) walks the single process-wide
_entries dict — every profile's keys — and finalizes expired sessions with no
scope installed, so _db resolved the ROOT store for rows that live under
profiles/<name>/state.db. The scoped inbound path and the unscoped background
path then maintained two copies of the same logical session whose end_reason
drifted apart independently. Once they disagreed, the #54878 stale-routing
guard read one copy while the routing index pointed at the other, and a live
conversation was dropped and recreated — silently, since that branch only sets
was_auto_reset when a reset policy also fired.
Field evidence from a live two-profile install: session 20260814_234313 was
end_reason=None in the root store but agent_close in the profile store, while
20260822_225807 was inverted. Both directions, which rules out a single
mis-scoped writer.
The owning profile is already encoded in the session key, so derive the store
from it: _profile_home_for_key / _db_for_key, plus _db_for_session_id for the
entry points addressed by session id. 40 self._db uses across 14 methods now
resolve that way. No signature changed and no existing test was modified.
_profile_home_for_key returns None when multiplexing is off, when the key
carries the legacy agent:main namespace, or when the profile has no live
directory, so single-profile installs resolve exactly where they always did.
The explicit-path branch still goes through SessionDB.__init__ ->
_ensure_test_isolation, keeping the live-DB guard over per-profile paths.
Part of #66887. The routing-index half — _routing_scope() and the sessions.json
mirror still pinned to one frozen sessions_dir while the handle moves — is left
for a follow-up rather than mixed in here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A non-admin '/sessions all' or '/resume --all' silently downgraded to
chat-scoped listing with zero feedback, which reads as 'my session
vanished' (community Telegram report). Both surfaces now append a notice
that cross-chat listing requires a configured admin. Follows up the
salvaged current-session '(current)' marker (PR #68556, fixes#68547):
sibling tests updated to pin the new contract, new i18n key
gateway.resume.all_requires_admin added to all 17 locales, docs updated.
Main removed the live-path runtime FTS rebuild (18ac3c4fb6) and #99652
narrowed the corruption classifier, after the salvaged PR #89364 branched.
Update its tests to assert the current contract: no fail-open detach on a
replaced file (fts_enabled stays True, no stale marker), fail-open (not
rebuild) for genuine same-file FTS corruption, and FTS-provenance errors
for the guard probe since generic malformed no longer reaches fail-open.
Detect same-inode cp via a generation stamp, halt FTS repair, and divert
unwritten transcripts to sessions/<id>.jsonl plus the gateway pending spool.
Co-authored-by: Cursor <cursoragent@cursor.com>
Addresses Enough1122's two non-blocking review notes on PR #92316:
1. Lease vs deadline arithmetic: the state.db init/migration/repair leases
(600-900s) are authoritative against the 300s default deadline by design.
Add explicit comments at all three lease sites pinning the trade-off:
single lease is deliberate (clamped to _MAX_LEASE_S=900), honest worst
case is up to the lease duration of zombie time on a wedged DB phase,
accepted over per-chunk renewal complexity in the migration loops.
2. Arm-site coverage: add a structural contract test asserting every
documented entry point (hermes_cli/main.py argv fast-path,
hermes_cli/gateway.py config-bridge re-arm, gateway/run.py backstop,
cli.py legacy --gateway) actually calls arm_startup_watchdog (or its
aliased import), so a future entry point can't silently ship unwatched.
Addresses the two class-level review blockers on PR #89750:
1. Bounded hard-exit seam (escort thread). The forensic fire path
(logger.critical, dump record, faulthandler, lifecycle ledger) can
itself wedge — the parked main thread may hold the logging handler
lock, or the disk may be full/hung. _fire() now starts an exit-escort
daemon thread BEFORE any forensics; it is free of log handlers,
filesystem access, module loads and application locks, and hard-exits
with the restart code after _FIRE_EXIT_BOUND_S unless the normal fire
path signals completion. Adversarial tests hold the logging handler
lock / hang the dump write at fire time and assert the exit seam is
still reached.
2. Phase-owned progress leases (report_startup_progress). Process CPU
time proves process activity, not startup progress: an unrelated busy
thread could extend forever while startup sits parked (false
negative), and I/O-bound repair/backup accrues ~zero CPU and would be
killed (false positive). Long synchronous startup phases now declare
authoritative, clamped (_MAX_LEASE_S), renewable progress leases:
state.db _init_schema + the version-gated data-migration chain
(hermes_state_schema) and repair_state_db_schema (hermes_state) are
wired. CPU progress remains only as a bounded fallback, capped at
_MAX_CPU_EXTENSIONS, with leases outranking the cap. Adversarial
tests cover both directions (lease saves zero-CPU legitimate work;
capped CPU noise no longer hides a parked deadlock).
Fire-path dump record now includes lease_count/last_lease_phase for
forensics. gateway/startup_watchdog.py shim re-exports
report_startup_progress.
OOF-298
Independent review of the initial startup-liveness watchdog surfaced two
P1s and three P2s. All are addressed here.
P1 — legitimate slow startups (large state.db schema migrations inside
SessionDB.__init__, which run synchronously before the loop starts) could
exceed the fixed 300s deadline and restart-loop. The watchdog now checks
process CPU time (time.process_time(), process-wide) when the deadline
expires: continuous CPU consumption means a live migration, so the deadline
is extended (with a warning log per extension). The OOF-298 deadlock class
parks every thread in futex waits and accrues ~zero CPU, so it still fires
on schedule. Documented limitation: a spinning busy-wait deadlock reads as
progress and won't fire — the observed incident class is parked threads.
P1 — import-time deadlocks were outside coverage. The implementation moved
to a stdlib-only top-level module (hermes_startup_watchdog), and
hermes_cli/main.py arms it via an argv fast-path ("gateway" + "run" in
argv) BEFORE the heavy module-level import graph. gateway/startup_watchdog
remains as a re-export shim so the intuitive import path keeps working for
the disarm site, tests, and REPL use. Import-lightness is a correctness
property, tested via AST inspection: at fire time the wedged main thread
may hold the import lock, so the fire path performs no imports on its own
thread — the lifecycle-ledger write runs on a bounded-join helper thread
and os._exit happens regardless.
P2 — disarm/fire race: the handle now has an explicit state machine
(armed → disarmed | firing) guarded by a lock; whichever transition takes
the lock first wins, so a disarm landing after deadline expiry but before
the fire transition is honored. Regression test forces the exact
interleaving by blocking inside the CPU probe.
P2 — uncovered entry points: cli.py --gateway and scripts/hermes-gateway
run_gateway() now arm the watchdog before importing the gateway graph.
hermes_cli/gateway.py run_gateway() keeps an idempotent backstop arm for
programmatic callers.
P2 — respawn-storm backoff interaction: the storm breaker's intentional
backoff sleep (up to minutes, ~zero CPU — indistinguishable from a parked
deadlock) now calls kick_startup_watchdog(extra_s=backoff) so the deadline
is pushed past the sleep instead of firing mid-backoff.
Also: the faulthandler stack dump is now additionally written to
logs/gateway-startup-watchdog.log (stderr may be absent on detached/
windowless runs); the disarm site in gateway/run.py moved inside the
loop-confirmed branch (if the loop is NOT live, the milestone was not
reached and the watchdog must stay armed); hermes_startup_watchdog added
to pyproject py-modules so sealed venvs ship it; SERVICE_RESTART_EXIT_CODE
is duplicated in the stdlib-only module with a parity test against
gateway.restart.
Tests: 38 in tests/gateway/test_startup_watchdog.py (contracts incl.
stdlib-only AST check and shim re-export identity, config resolution,
arm/disarm/kick, CPU-progress extension vs no-progress fire, probe-failure
fails toward firing, disarm-vs-fire race, dump record + file stacks,
lifecycle ledger, custom exit code).
A hosted gateway (hermes-doubleam-2568) deadlocked at startup with every
thread parked in futex_wait_queue before the asyncio loop came alive:
zero log lines, /health unreachable — but s6 saw a live PID so it never
respawned the process, and a stale gateway_state.json from the previous
life told every status surface "draining" for ~30 hours.
Every existing liveness backstop assumes startup succeeded: the
loop-liveness watchdog is armed inside the running loop's startup path,
the shutdown watchdog arms at stop(), and the heartbeat file is written
by an asyncio task. None can fire when the process wedges before the
loop exists.
New gateway/startup_watchdog.py: a plain daemon OS thread armed at
process entry (both gateway.run.main() and the `hermes gateway run` CLI
wrapper), disarmed the moment GatewayRunner confirms a live event loop —
the point where the existing loop-liveness watchdog takes over. If
startup neither reaches that milestone nor exits within the deadline
(default 300s; slowest legitimate pre-loop work is the 120s-bounded MCP
discovery wait), the watchdog:
* dumps all-thread stacks via faulthandler,
* appends a JSON record to logs/gateway-startup-watchdog.log,
* records the exit in the NS-608 lifecycle ledger
(reason=startup_liveness_watchdog) so the next boot classifies it
instead of reporting an unclean SIGKILL/OOM death,
* os._exit(75) so s6/systemd respawn the process.
Config is env-only (HERMES_STARTUP_WATCHDOG=0 to disable,
HERMES_STARTUP_WATCHDOG_TIMEOUT_S to tune, floor-clamped to 30s):
the watchdog must be armed before config.yaml is loaded — a wedge
during config parsing is exactly in scope — so it cannot depend on
config for its own enablement. Everything is best-effort; a watchdog
failure never affects the startup it observes.
Arm sites are placed after the PID-file/--replace conflict guards so a
--replace loser exiting early never arms a watchdog. Disarm happens even
when the loop guards are config-disabled (gateway.loop_watchdog: false)
— the startup watchdog only covers the pre-loop window, never adapter
connects or steady-state, so WhatsApp pairing / npm cold installs are
unaffected.
Tests: tests/gateway/test_startup_watchdog.py (29 tests — config
resolution, arm/disarm idempotency, fire path with captured exit,
lifecycle-ledger marking, dump record, disable knob).
Fixes OOF-298.
When the default/global state.db is corrupt at gateway startup,
SessionStore degrades (_db=None) and record_gateway_session_peer never
self-heals. Under multiplexed profile routes the AIAgent's lazy
_ensure_db_session was then the ONLY durable write for the session, and
it created the row identity-less (session_key/chat_id/chat_type/
thread_id/user_id/origin_json all NULL) — unrecoverable by
find_latest_gateway_session_for_peer, so Telegram chats forgot prior
turns.
Persist the routing identity the agent already carries into
create_session; plain CLI sessions keep the old identity-less shape.
Salvaged leg 1 of #88804; the transcript-recovery leg is covered by the
scope-aware session DB resolution already on main.
Follow-ups on top of the salvaged commits from PR #87111 (@HexLab98) and
PR #87265 (@JoaoMarcos44):
- keep main's #92991 stall watchdog (150s progress-based) as the single
steady-state liveness probe instead of adding a second overlapping one
- orphaned-client aclose() cleanup uses the wall-clock thread deadline and
is tracked in _background_tasks so a wedged close can neither hang nor
leak one task per reconnect attempt (from #87265's review findings)
- merge #87265's no-keepalive getUpdates pool (max_keepalive_connections=0)
with #87111's TCP-keepalive socket options on all transports
- add tests/gateway/test_telegram_closewait_windows_live.py: live probes
against a real half-closing HTTP server, skipif non-win32, wired into
the on-demand windows-venv-e2e lane (wine2e/**)
Prevent the dedicated getUpdates pool from reusing server-closed connections and replace a polling HTTP client left open after a timed-out CLOSE-WAIT drain. Keep the general Bot API pool reusable so concurrent sends and edits are unaffected. Add regression coverage for both transport limits and stale-client replacement. Fixes#87057
shutdown_mcp_servers() blocks on future.result(timeout=15) which, called
from the gateway event-loop thread during SIGTERM teardown, freezes the
loop for up to 15s when the MCP loop and its stdio children are torn down
concurrently. Supervisors with a shorter kill grace (s6-overlay: 3s)
SIGKILL the gateway before lifecycle_ledger.mark_exited() runs, producing
phantom 'exited UNCLEANLY' reports on every subsequent boot.
Run the sync shutdown on a daemon thread and poll via _await_thread_exit
with a 5s budget; proceed with teardown if it wedges. Fixes#82874;
completes the shutdown half of #64155.
Widen #96038's fail-closed classifier to the gateway transcript retry
path: SessionStore._is_fts_corruption_error no longer treats a generic
'database disk image is malformed' as FTS-only damage. It now delegates
to SessionDB._is_fts_write_corruption_error (SQLITE_CORRUPT_VTAB result
code or explicit fts5 corrupt-structure text) and only keeps the
messages_fts-named cases. Structural corruption falls through to the
bounded retry/backoff path instead of rebuilding FTS and retrying writes
against a damaged database.
Sibling site spotted in PR #98090 by @fangliquanflq.
Issue #98573 reports a long-lived gateway holding ~20 `state.db` descriptors
that never shrink, walking into the 256 soft RLIMIT_NOFILE a launchd/systemd
service manager hands the process. The cause named there — a per-thread
`threading.local()` read connection — is already gone (87aedbe7b6 pooled the
read connections, 0472c31aa1 added the peak permit). Measured on main: one
SessionDB with 40 concurrent reader threads peaks at 9 live connections, not 40.
The symptom survives one layer up. `_READ_POOL_MAX` was enforced by a
BoundedSemaphore owned by each SessionDB, which bounds the wrong noun: the
descriptors are spent on a FILE, so every additional handle on one state.db got
its own allowance and peak scaled as `instances x (1 + _READ_POOL_MAX)`.
Two changes, both needed:
* The permits move to a per-path `_PathReadBudget`, shared by every SessionDB
in the process that points at that file. A permit miss first reclaims an IDLE
pooled connection from a peer handle before degrading to the writer lock —
without that, whichever handle warmed up first would pin the whole budget and
permanently demote every later one (a cron job's transient handle, a second
profile's store) to the locked writer connection.
* `GatewayRunner` borrows `SessionStore`'s handle instead of opening its own.
Both caches resolve the same `_default_db_path()`, so the process was holding
two writer connections and two read pools against one file for no reason, and
doubling again per profile on a multiplexed gateway. The store owns the
connection and sweeps it at shutdown; the runner's cache now holds only the
async wrapper and its sweep skips borrowed handles.
Measured peak live connections against one file, 40 reader threads, by handle
count 1/2/4/8:
before: 9 / 18 / 36 / 51 (51 not 72 only because the sample window ended
before every pool filled)
after: 9 / 10 / 12 / 16 (read connections capped at 8 in total; the
remainder is one writer per handle, and the
gateway's per-profile pair is now one)
Fixes#98573
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adapt five scenarios from @liuliu0223's regression suite in #99021:
- paired-mode (empty allowlist) positive paths for approval and
update-prompt cards, the DM breakage this fix resolves
- fail-closed rejection of clicks with an empty operator identity
- chat-mismatch rejection when an approval card is forwarded
The synchronous card-action handlers and the update-prompt resolver
authorized clicks with _allow_group_message(), which answers "may this
sender chat in this group?" — with group_policy=open it returns True
for everyone. The approval resolver already used the correct operator
gate (_is_interactive_operator_authorized), so the three code paths
disagreed: with an open group policy an out-of-allowlist click on an
update-prompt card was fully executed, and approval clicks returned a
resolved-looking card before being rejected asynchronously.
Authorize all three paths with _is_interactive_operator_authorized(),
which checks membership of admins ∪ allowed_group_users (wildcard and
the empty pairing-mode allowlist keep their existing allow semantics,
matching _admit's DM pairing default). A missing operator identity now
fails closed on the update-prompt resolver instead of skipping the
check.
Fixes#96045
The orphan-backend classifier fingerprinted candidates via
gateway.status.get_process_start_time, which prefers /proc/<pid>/stat —
the HOST process table, in clock ticks. Under the fake-psutil test harness
(and any containerized run where the PID number happens to exist on the
host) that returns the WRONG process's fingerprint in the WRONG units,
while pid_is_hermes verifies via psutil centiseconds at kill time: the
guard would then refuse every legitimate reap. Read create_time() from the
same psutil handle used for classification, quantized exactly like
gateway.status does on Windows, so the fingerprint round-trips.
Also covers the Windows-lane sibling: test_uses_netstat_and_taskkill_on_windows
now pins the guarded call path, plus a new refusal test for a non-bridge
listener PID (#89614 class).
Salvage hardening on top of the three cherry-picked contributor commits
(#91297 gebilaowang404 + AlexMnrs, #96741 burak33bb, #98826 ayushnangia),
closing the remaining unverified-PID kill sites as one class (#98814, #89614):
- pid_is_hermes: token-boundary 'hermes' match (no more loose substring
false-positives), and an explicit start-time expectation is now honored
on POSIX too (a mismatched fingerprint is a recycled PID on any platform).
- kill_process_tree: drop the guard on our OWN retained Popen child — a
retained handle pins the PID, so the check could only false-refuse.
- gateway.status.terminate_pid: POSIX force-kills also refuse when a
caller-provided expected_start_time no longer matches.
- kill_gateway_processes: re-verify the LIVE cmdline at kill time (the
scan-time match is a TOCTOU window).
- _reap_unsupervised_gateway_orphans: fingerprint orphans at scan time and
require a still-matching identity before the delayed SIGKILL escalation.
- whatsapp _kill_port_process: never kill a bare netstat/lsof-scanned PID
unless the live process is actually a node bridge (was a stranger-kill).
- browser daemon reap/close paths: pass the start-time fingerprint into
ProcessRegistry._terminate_host_pid (previously unverified), and the
session-close path now runs the same daemon identity verification as
the orphan reaper.
- tests/hermes_cli/test_taskkill_identity_windows_live.py: live Windows
probes (real spawned processes, real psutil ancestry) wired into the
on-demand windows-latest wine2e lane.
Fixes#98814Fixes#89614
Post-rebase composition over #99431/#99429/#99427: file-attachment sends
route through _run_message_send so the mention-recovery ladder covers
media captions; _send_file_attachment/_send_local_file honor the
resolved thread-root anchor and reply_to_mode opt-out; send() records
event_meta on the verified receipt id (#75826); test fakes gain the
auth_tag kwarg and accepted-receipt shape.
Localizing inbound relay media spends the agent's own Buzz credentials on
a URL chosen by the sender, so it must not run on the strength of the
adapter's local allow-list alone. Require the gateway's authorization
callback to return an explicit True before any `buzz media get` runs; a
denial, a missing callback, or a raising callback fails closed and leaves
the message text exactly as it arrived.
`_is_sender_authorized` previously wrapped the callback result in
`bool()`, so a truthy non-boolean (a status string, a sentinel) would
satisfy an `is True` gate's intent while bypassing its guarantee. Only
the literal booleans now propagate; anything else is "unknown", which the
existing Slack and Discord callers already treat as trust-unknown.
Reviewers asked for this boundary on the sibling inbound-media PRs
(#77734, #78051); it applies equally to the retrieval path in #75614,
which this change builds on.
kshitijk4poor