6 Commits

Author	SHA1	Message	Date
Teknium	cfa327e5dc	refactor(hclib): remaining hermes_cli library modules — dead code, unified helpers, flattened branches	2026-09-02 15:03:42 -07:00
Jeremy	a266155cc4	fix(cli): untrack sqlite connections only after close succeeds ... A failed close left the FD open while the byte-probe guard thought nothing was live. Keep the registry entry until close actually works.	2026-07-31 23:18:12 -07:00
teknium1	9657f6e343	fix(sessions): close the snapshot check/use race and guard damaged state_meta ... Post-merge follow-up to #71770. Both defects were found by @helix4u in review and reproduced against merged main before fixing. 1. Check/use race in _copy_source_bundle (my bug, from the #71770 follow-up commit). It called has_live_connection(), released the registry lock, and only then ran shutil.copy2() over the bundle. A tracked connection could open in that window; the copy's close() then cancels its POSIX advisory locks -- the exact class #71724 closed. Measured on main: a racer thread opened a connection mid-copy after blocking 0.000s. Adds sqlite_safe_read.offline_file_access(), a context manager that holds the connection-lifecycle lock across an entire multi-step raw access, and routes the bundle copy through it. Same racer now blocks 10.0s until every raw descriptor is closed. Any future raw read of a database file (hashing, moving a bundle aside) should use this rather than a bare pre-check. 2. _copy_state_meta_salvage assumed a 'key' column. A damaged state_meta can keep 'value' and lose 'key'; columns.index("key") then raised ValueError and aborted the whole partial recovery. The mirror case (key without value) would have copied key-only rows and reported the table complete. Now requires both, matching the non-partial _copy_state_meta, so an unusable optional table is recorded as missing/failed and --allow-partial still recovers sessions and messages. Both regression tests verified by sabotage: reinstating the bare pre-check fails the race test, removing the key/value requirement fails the other. 937 targeted tests green.	2026-07-25 23:05:30 -07:00
teknium1	fe431651c5	fix(state): make the byte-probe guard atomic, path-correct, and fail-closed ... Addresses review findings on the previous commits. Three of them were real defects I reproduced against my own head before fixing. 1. Check/use race (BLOCKING). read_header_bytes_preopen() checked has_live_connection() under _live_lock, released it, then did the raw open/read/close outside the lock; connect_tracked() opened before registering. A thread could pass the "nothing is live" check, another could open a connection and BEGIN IMMEDIATE, and the first thread's close() then cancelled its POSIX locks -- the exact bug this guard exists to prevent. Reproduced deterministically (BLOCKED -> ACQUIRED). _live_lock now spans all three lifecycle transitions: open+register, unregister+close, and check+open+read+close. 2. Read-only connections keyed by URI spelling (BLOCKING). SessionDB's read-only path opens file:/…/state.db?mode=ro; that string was fed to Path.resolve(), producing <cwd>/file:/…/state.db?mode=ro. No probe of the real Path could match, so read-only connections were invisible to the guard and their locks cancellable. Reproduced with no forced scheduling. Keys now come from PRAGMA database_list (canonical path), with an explicit tracking_path override. 3. Fail-open wrapper (HIGH). _connect_tracked_db() caught every exception and retried an untracked plain connect, so any error silently disabled the guard. Now only ImportError (scaffold installs without hermes_cli) falls back; real failures propagate. 4. Backup paths that warned and proceeded (MEDIUM). _backup_corrupt_db() and _backup_db_file() raw-read live databases; they now REFUSE when a connection is live rather than warning. Losing a forensic copy beats corrupting the database being rescued. Custom factories are no longer rejected (that broke legitimate callers) nor silently untracked -- the tracking close() is mixed into whatever factory is in play, including when an opener substitutes its own after the fact. WAL POLICY: #70055 is RESTORED, not reverted. My earlier justification was confounded -- the clean WAL result came from 3.53.1, which carries both the WAL-reset fix AND 3.51.0's broken-lock defenses, so it said nothing about the bundled 3.50.4. Re-measured on 3.50.4 with the lock fix in place: WAL 0/3 and DELETE 0/3, i.e. no evidence WAL is safer. Upstream still documents the WAL-reset bug through 3.51.2 as serious. Keeping new databases out of WAL until a fixed runtime ships is the conservative call, and the WAL policy does not belong in this root-cause fix. Six sabotage runs confirm each new test fails when its defect is reinstated (including two that initially did NOT -- the race test was rewritten to pause inside the byte read, and a separate test added for the opener-substituted factory path). 1124 targeted tests green.	2026-07-25 21:44:43 -07:00
teknium1	95fb477856	fix(state): close the tracking leak and finish the audit of raw DB reads ... Completeness pass over the previous commit. Registry leak (would have silently disabled the guard): the first version incremented on open but decremented only in SessionDB.close(). kanban's connect() hands raw connections to callers who close them directly (4 sites), so its counter only ever went up — after enough kanban operations every byte-probe on that path would be refused forever, disabling zeroed-file and header detection. Replaced manual track/untrack with a TrackedConnection subclass that untracks in close(), the one method every close path goes through. Verified across plain close, contextlib.closing, double close, nested lifetimes, and 100-cycle churn; `with conn:` (a transaction scope, not a close) correctly stays tracked. Also: a caller-supplied factory now wins instead of raising TypeError on a duplicate kwarg, since tracking is an optimisation for the probe guard, not a precondition for opening the database. Removed _apply_delete_for_wal_reset_bug, dead after the force-DELETE revert. Audit notes: - DELETE mode is still reachable via the NFS/SMB/FUSE fallback and remains correct there; those users are protected by the raw-read fix, not by the journal mode. - The remaining whole-file reads are on genuinely offline artifacts: _backup_db_file (DB won't open; bytes preserved for forensics), _backup_corrupt_db (quarantine path, now warns if a connection is live), and backup verification of snapshots. backup._safe_copy_db already uses the SQLite backup API rather than a byte copy. - The mechanism is POSIX-specific (Windows byte-range locks are handle-scoped, not process-scoped), so this is a Linux/macOS correctness fix; the change is platform-neutral and safe on Windows. Sibling tests updated: session recovery no longer expects a DELETE-mode recovered DB, and the kanban WAL-fallback test patches the connect site that actually runs now.	2026-07-25 21:44:43 -07:00
teknium1	fbd5e5772b	fix(state): stop cancelling our own POSIX locks on live SQLite databases ... `hermes sessions optimize` could corrupt state.db. Root cause is Hermes, not the SQLite WAL-reset bug (#69784). close() on ANY file descriptor for a SQLite database cancels every POSIX advisory lock the process holds on that file, including a running VACUUM's EXCLUSIVE lock (sqlite.org/howtocorrupt.html section 2.2). Hermes byte-probed live databases in several hot paths: the zeroed-state.db detector runs on every SessionDB construction (and the gateway builds those constantly), and kanban's post-commit invariant check ran after every COMMIT. While VACUUM rewrote the file, those probes dropped its lock and let other processes write into it. A/B against the real code, only variable being the raw read: SQLite 3.50.4, VACUUM + concurrent writers, DELETE mode raw open/close during VACUUM 8 vacuums, 319 vacuum errors, 2/2 corrupt no raw read (control) 229 vacuums, 0 vacuum errors, 0/2 corrupt SQLite 3.53.1 (WAL-reset FIXED) reproduces identically: 2/2 corrupt. After this change: 0/4 corrupt, 0 vacuum errors. Because the upgraded runtime corrupts too, replacing the embedded SQLite does not fix this class; and because DELETE is where it reproduces, #70055's "force DELETE on vulnerable builds" mitigation steered users into the failing mode. That gate is reverted here: vulnerable builds get WAL again and still warn so operators can upgrade. - add hermes_cli/sqlite_safe_read.py: read page_count via PRAGMA over the existing connection instead of open()+seek(28); byte-level probes are restricted to before any connection exists and refused once one is live, with an explicit force= escape for offline artifacts (snapshots, archives) - track live connections in SessionDB and kanban's connect so that guard is enforced rather than merely documented - kanban's torn-extend check now only applies under a rollback journal; in WAL a committed page may still legitimately sit in the -wal file - revert the force-DELETE WAL gate and update the tests that pinned it Regression tests assert the behavioural contract (an external process stays locked out across Hermes' inspection calls) and were verified to fail when the old raw-open behaviour is restored.	2026-07-25 21:44:43 -07:00