- capture_retired_wal_generation: drop the main_image_max_bytes kwarg (no caller passes it; read the module constant).
- replace _write_json_durably with utils.atomic_json_write (late import keeps the capture dependency-light).
- clean the abandoned .partial staging dir on capture failure instead of leaving it for the retry to work around.
- _disable_close_time_checkpoint: document that it is the per-instance twin of _close_time_checkpoint_configurable and must agree with __init__'s capability decision.
- _close_quietly kept on the late import (module-level import from hermes_state_dbfile risks future cycles).
- hermes backup excluded *.db-wal by suffix but not the retired-wal capture dirs, so it would ship the capture's main-image copy while dropping the captured -wal that is the artifact's point; exclude <name>.retired-wal-* dirs whole (they must move as manifest+image+wal unit).
- _copy_range treated a short read as success, yielding a truncated copy with a valid manifest while the unlinked inode still dies at exit; raise RetiredGenerationCaptureError and clean the .part file.
- _quarantine_reason docstring no longer claims close() checks replaced before generation loss (close evaluates loss first and skips quarantine when lost).
_copy_descriptor and _copy_main_image were the same 17-line pread/sha256/fsync/rename loop
differing only in the byte source; both now delegate to _copy_range(read, dest, size=).
The header parser reuses _STATE_DB_APPLICATION_ID_OFFSET instead of re-spelling 68, and
_fsync_path keys off the module's _IS_WINDOWS like the rest of hermes_state_dbfile.
With the retired generation captured durably, closing a lost-generation
handle is safe wherever SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE took effect: the
frames are preserved and sqlite3_close no longer checkpoints them into the
newer generation. On Python 3.11, where sqlite3 has no setconfig, closing
still runs SQLite's internal checkpoint over the newer main file, so only
there the exact quarantined connection is retained instead of closed: one
public Py_IncRef reference via ctypes.pythonapi (no struct-layout access),
bound before the writer opens, taken in close() after the capture. Runtimes
with setconfig never touch ctypes; a writable SessionDB requires CPython
with ctypes only where retention is the sole guard.
Read-only handles and every other quarantine reason close as before.
Regressions cover both branches: where retention applies, the retired inode
stays readable after close() and GC and an independent deleted WAL under
the same pathname is untouched; elsewhere the capture is the surviving copy.
A separate process's newer generation survives close, GC and normal exit
in both cases. The mock-based close-time regression originally written for
Refs #105670
Co-authored-by: fangliquanflq <fangliquan@qq.com>
After DeletedWalGenerationError the retired frames exist only in an
unlinked -wal inode that this process keeps open. #106315 stops them from
being checkpointed under wrong page numbers, but the canonical remediation
("stop the gateway, dashboard and cron writers, then reopen") lets the
kernel drop the inode with them: committed transactions whose disposition
is unknown were destroyed by the prescribed recovery path itself.
Capture the exact retired generation next to the database at the first
halt, or at close() if that sees the loss first. The WAL is located by the
(st_dev, st_ino) recorded at open among this process's own descriptors,
never by pathname, so a sibling's deleted WAL or a newer sidecar minted at
the same path cannot be mistaken for it; it is read with pread and no
descriptor is closed, moved or truncated. The artifact holds the WAL, the
-shm when still ours, the main image (or its header past a size cap) and a
manifest with identities, digests and the sidecar generation found at the
path at capture time. Nothing is merged: whether the frames belong on top
of the file now at the path stays an operator decision. close() refuses to
settle without the capture: it raises and leaves the handle open.
Regressions: capture at halt and at close, refusal to settle on capture
failure, inode-not-pathname selection with a second deleted WAL under the
same name, refusal to guess without a recorded identity, and a subprocess
control where rows committed only in the retired WAL are recovered from
the capture alone after the writer process has exited.
Refs #105670
`is_zeroed_state_db` and `has_invalid_sqlite_header_preopen` shared the
is_file / stat / live-connection / read_header preamble; `_preopen_header`
owns it now (zeroed is the NUL subset of "no SQLite header"). The
quarantine message also names `hermes sessions recover --source <bak>` so
the preserved bytes are actionable, not just parked.
Startup only quarantined a 0-byte / all-NUL state.db. A file whose first page
was clobbered with record bytes (#102198) went straight to sqlite3.connect,
which raised "file is not a database" and deleted the -wal sidecar — the one
piece of evidence that could have been recovered.
`has_invalid_sqlite_header_preopen` generalises the zeroed probe (zeroed is a
subset of "no SQLite header"; same live-connection contract, never raises).
`quarantine_invalid_state_db` moves the file AND its -wal/-shm aside as
`state.db.<zeroed|notadb>-<ts>-<pid>.bak` before anything opens it; a fresh
DB is created as before.
Re-authored on current main (the quarantine helpers moved to
hermes_state_dbfile.py in d15c61b5dc); one regression test proves the
notadb case is quarantined with its sidecars and the new DB passes
integrity_check.
Refs #102198 (the write-after-SIGTERM that clobbers page 0 is not addressed
here; this preserves the evidence instead of destroying it).
hermes_state.py: delete every '# noqa: F401 (re-exported...)' import block (hermes_state_common/errors/guard/
readpool/sessions/fts/dbfile/wal/repair/registry + agent.context_compressor _DB_PERSISTED_MARKER_KEY); keep
only the names hermes_state.py itself uses, without noqa.
hermes_state_registry.py: drop get_shared_session_db/release_shared_session_db/close_shared_session_dbs
aliases; every caller (gateway/, tools/, tui_gateway/, cron/, mcp_serve, run_agent, tests) now imports
acquire/release/close_all/release_or_close from hermes_state_registry.
hermes_state_titles.py: drop set_auto_title_if_empty shim (title_generator keeps its getattr fallback).
Re-remove shim-only names restored by 34abf954bd: latest_user_message_row_id (tests call
latest_message_row_id(key, role='user'); role-targeting assertions kept) and get_session_activity (tests
build the snapshot via agent.session_activity.build_activity_snapshot over db.get_session(sid)).
hermes_state_wal._log_once resolves its dedupe sets as module globals instead of via hermes_state;
hermes_state_repair helpers call module globals directly (tests patch hermes_state_repair.<name>).
Frozen updater surface untouched (update_cmd_maint imports only SessionDB from hermes_state).
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
hermes_state_maintenance.py 557->418, hermes_state_dbfile.py 545->404.
- _placeholders() replaces 4 inline ','.join('?'...) builders (identical output)
- _write_guards_reject() unifies the lease/lock probe in sweep_orphaned_sessions
and prune_sessions (same kwargs, same exception set; prune keeps set -= order)
- _page_pragmas() absorbs the try/except-debug shape of logical_size_bytes and
_freelist_ratio (log texts unchanged); _try_checkpoint() for the two WAL
checkpoints in vacuum(); _seconds_since() for the two state_meta float parses
- archived tri-state -> f'string' clause (byte-identical SQL)
- dbfile: contextlib.suppress for pass-only excepts, lock closures collapsed,
unreachable size<0 branch dropped, is_zeroed tail folded to one predicate
- docstrings/comments compacted by hand; every WHY/lock-safety invariant kept
SQL PARITY OK (1120 stmts), MSG PARITY OK, import smoke OK.
kshitijk4poor