The Aug 12 pin bump (91345435a) updated pyproject/uv.lock but not
tools/lazy_deps.py, tripping the #31817 downgrade-guard tests; the
post-VACUUM TRUNCATE fix landed without updating the checkpoint test
that pinned the old no-TRUNCATE rule. Both red on pristine main.
SessionDB.close() ran `PRAGMA wal_checkpoint(TRUNCATE)`. Every cron
run_agent opens and closes its own transient SessionDB, so on a busy
fleet this fired a full WAL reset many times an hour, racing the
gateway's long-lived writer on a large WAL database and tearing hot
B-tree pages -- structurally the same corruption this module's own
periodic checkpoint was already switched to PASSIVE to avoid (#45383).
Only close() and two manual-maintenance paths still used TRUNCATE.
Route every checkpoint on the shared state.db through PASSIVE:
- close() (hermes_state.py)
- pre-VACUUM in vacuum() (hermes_state.py)
- post-optimize-storage (hermes_state_search.py)
PASSIVE never resets/truncates the WAL and never takes the exclusive
checkpoint lock, so it cannot lose a transient closer's race with the
live writer. The WAL is instead bounded by `journal_size_limit` and the
writer's natural post-checkpoint reset. TRUNCATE belongs only on a
sole-opener/quiescent connection (e.g. offline maintenance); this change
does not try to detect that -- PASSIVE is the safe default.
Diagnosed as the root cause of three state.db B-tree corruptions in
2026-08: damage localized to the hottest-written pages (gateway_routing
and the sessions indexes), with whole zero-filled pages still live and
off the freelist -- the checkpoint/reset-race signature, not disk or
application SQL.
Tests: tests/test_wal_checkpoint_strategy.py now asserts PASSIVE at
close(), before vacuum(), and after optimize_fts_storage() VACUUM;
tests/test_hermes_state.py asserts close() likewise. Focused run:
226 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TRUNCATE checkpoint every 50 writes causes B-tree corruption on large
databases (65K+ pages) due to the exclusive-lock I/O pressure from
checkpointing thousands of frames at once.
Switch periodic _try_wal_checkpoint() to PASSIVE mode which does not
require an exclusive lock and cannot corrupt pages under I/O pressure.
Keep TRUNCATE in close() and pre-VACUUM paths where it is safe
(infrequent, controlled conditions).
Also replace silent `except Exception: pass` with logged warnings for
checkpoint failures so operators can detect early corruption signals.
Fixes#45383
Teknium