fix(state): consolidate gateway SessionDB writers via process-wide shared registry

A gateway process opened state.db from ~12 call sites, each minting its
own writer connection, self._lock, close-time WAL checkpoint, and
token-writer thread. With N independent writers on one WAL file, one
connection's close-time checkpoint could race another's growth — the
lost/reordered-page-write signature across 11+ incidents (#90837).

Adds hermes_state_registry.py: a process-wide, per-path, refcounted
shared registry owning the writer boundary.

- acquire(path): same resolved path returns the same instance (one
  writer connection, one lock, one token-writer thread) for every
  long-lived in-process caller (gateway runner, SessionStore, per-agent
  lazy recall, cron per-job, mirror, channel_directory, slash_commands,
  shutdown_flush, session_search, react_to_message, delegate, mcp_serve,
  auto_archive, tui_gateway).
- close() on a shared instance is a NO-OP — the registry owns the
  lifecycle, so one caller's close can never tear down a writer other
  callers still hold.
- Generation-aware retirement on inode change: a replaced state.db
  RETIRES the live generation (never lent again) but keeps it alive for
  existing holders; release is object-keyed so holders of the old
  generation drain it independently of the new one. The old
  generation's own write path still fails with the typed
  StateDbReplacedError (existing protection, unchanged).
- Replacement-open failure leaves NO registry entry for the path —
  the next acquire retries fresh, never hands out a closed stale object.
- All teardown runs OUTSIDE the registry lock: a final release's WAL
  checkpoint can never stall acquisition for every state.db.
- close_shared_session_dbs() at gateway shutdown drains every
  generation (live + retired) as the final safety net.

CLI one-shots, recovery flows, and read-only cross-profile opens keep
using SessionDB() directly with their own close() — only long-lived
in-process sites route through the registry.

References #90837 (root-cause tracker stays open: the #10 EOF signature
and the WAL-lifecycle A/B verdict remain under investigation there).
This commit is contained in:
kshitijk4poor
2026-09-01 20:20:56 +05:30
committed by kshitij
parent e2521fdef6
commit db339f0051
39 changed files with 892 additions and 148 deletions
+6 -3
View File
@@ -679,9 +679,9 @@ class AIAgent:
if self._session_db is not None:
return self._session_db
try:
from hermes_state import SessionDB
from hermes_state import get_shared_session_db
self._session_db = SessionDB()
self._session_db = get_shared_session_db()
# We opened it here, so nothing else holds a reference — this agent
# is its only owner and close() must release it.
self._owns_session_db = True
@@ -5081,7 +5081,10 @@ class AIAgent:
try:
if getattr(self, "_owns_session_db", False) and session_db is not None:
self._owns_session_db = False
session_db.close()
# Shared instances no-op on close(); release the refcount
# so the registry can close when the last caller is done (#90837).
from hermes_state import release_or_close
release_or_close(session_db)
except Exception:
pass