8 Commits

Author	SHA1	Message	Date
Teknium	53db597201	simplify(compat): hermes_state — drop 81 re-exports + 3 registry aliases + 3 shims, repoint 45 callers + 60 test files ... hermes_state.py: delete every '# noqa: F401 (re-exported...)' import block (hermes_state_common/errors/guard/ readpool/sessions/fts/dbfile/wal/repair/registry + agent.context_compressor _DB_PERSISTED_MARKER_KEY); keep only the names hermes_state.py itself uses, without noqa. hermes_state_registry.py: drop get_shared_session_db/release_shared_session_db/close_shared_session_dbs aliases; every caller (gateway/, tools/, tui_gateway/, cron/, mcp_serve, run_agent, tests) now imports acquire/release/close_all/release_or_close from hermes_state_registry. hermes_state_titles.py: drop set_auto_title_if_empty shim (title_generator keeps its getattr fallback). Re-remove shim-only names restored by 34abf954bd: latest_user_message_row_id (tests call latest_message_row_id(key, role='user'); role-targeting assertions kept) and get_session_activity (tests build the snapshot via agent.session_activity.build_activity_snapshot over db.get_session(sid)). hermes_state_wal._log_once resolves its dedupe sets as module globals instead of via hermes_state; hermes_state_repair helpers call module globals directly (tests patch hermes_state_repair.<name>). Frozen updater surface untouched (update_cmd_maint imports only SessionDB from hermes_state).	2026-09-03 13:46:50 -07:00
HexLab98	c5138618f7	test(state): cover deferred FTS retry, leftover lock files, and WAL interpreter identity	2026-09-02 04:15:02 -07:00
Teknium	c4aea32317	fix(db): never downgrade journal mode on a database with concurrent openers ... The WAL-reset-vulnerability gate (#70055 lineage) could flip a LIVE WAL database to journal_mode=DELETE while another process was writing to it. Observed on state.db (Aug 5): a pytest process on the repo .venv (SQLite 3.50.4, vulnerable) opened the live ~/.hermes/state.db while the gateway (SQLite 3.53.1, WAL) held it, downgraded the journal mode, and destroyed the gateway's committed-but-uncheckpointed WAL transactions (disk rows went 10 -> 0 while memory held 185). cron/executions.db already had the "leave WAL in place, no live downgrade under concurrent openers" rule via the on-disk WAL probe; state.db and every other store shared the hole whenever the mode PROBE itself was blocked by a concurrent opener's locks ("could not read the mode" was treated as "not WAL" -> flip anyway). Generalized in the single journal-mode owner (apply_wal_with_fallback), covering ALL call sites (state.db, kanban.db, projects.db, cron/executions.db, delivery_ledger, async_delegation, verification_evidence, discord recovery, response_store.db, memory_store.db): - _set_journal_mode_no_wait(): the only journal-mode switch primitive for non-WAL targets. Forces busy_timeout=0 around the pragma so SQLite's own exclusivity requirement for leaving WAL becomes the concurrent-opener detector — any other opener (this process or another) makes the flip fail immediately instead of waiting out a busy timeout and sneaking the flip in under a live writer. - Vulnerable-SQLite gate: an unreadable journal mode (probe blocked) now means "ownership not provably exclusive" — leave the mode untouched and warn, never flip. A lock conflict on the flip itself likewise leaves the mode alone. - Configured journal_mode=delete: refuses (raises) rather than downgrading blind when the mode cannot be verified under a concurrent opener. - Filesystem-incompat fallback: re-raises instead of downgrading when the on-disk mode cannot be verified. - New/exclusively-owned DBs on vulnerable builds behave exactly as before (DELETE gate retained per #70055). Behavioral tests use a REAL second process (and a real second connection holding an exclusive lock) with the blocked-state assertions running WHILE the holder owns the DB, plus exclusive-ownership downgrade-still-happens coverage.	2026-08-06 07:49:42 -07:00
Teknium	6b81590c55	test: prune low-value tests suite-wide (wave 1) — 46,820 → 28,106 test functions ... Systematic prune per AGENTS.md test policy, one pass over every major test tree (gateway, hermes_cli, tools, agent, run_agent, plugins, cli, cron, tui_gateway, honcho/openviking, root-level): - DELETE: source-reading tests (read_text/getsource on prod files), change-detector tests (exact catalog counts, model-name snapshots, config version literals), mock-echo tests (assert a mock returns what it was told), assertion-free/trivial tests, near-duplicate parametrizations (boundaries + one representative kept), async/sync twin duplicates, cosmetic within-file variations. - KEEP (mandatory): security/redaction/approval guards, message-role alternation invariants, prompt-caching/deterministic-call-id invariants, issue-number regression tests (deduped), E2E tests. - 6 test files deleted outright (script-style/no-assert or fully redundant); conftest.py, fakes/, fixtures/ untouched. - tests/acp/conftest.py added: autouse fixture stubs the live models.dev/GitHub/Copilot/Anthropic inventory fetches that ACP server tests performed on every session create — test_server.py 147s → 3.4s, and the tests are now genuinely hermetic. - Sleep-based slowness shrunk where safe (codex_ttfb_watchdog, compression_concurrent_fork, etc.); no wall-clock assertion tightened. Verification: full hermetic suite via scripts/run_tests.sh — 2439 files, 31,130 tests passed, 0 failed, 0 flaky retries, 315s wall (baseline: 583s wall, 13,564s subprocess CPU).	2026-07-29 13:10:23 -07:00
teknium1	fe431651c5	fix(state): make the byte-probe guard atomic, path-correct, and fail-closed ... Addresses review findings on the previous commits. Three of them were real defects I reproduced against my own head before fixing. 1. Check/use race (BLOCKING). read_header_bytes_preopen() checked has_live_connection() under _live_lock, released it, then did the raw open/read/close outside the lock; connect_tracked() opened before registering. A thread could pass the "nothing is live" check, another could open a connection and BEGIN IMMEDIATE, and the first thread's close() then cancelled its POSIX locks -- the exact bug this guard exists to prevent. Reproduced deterministically (BLOCKED -> ACQUIRED). _live_lock now spans all three lifecycle transitions: open+register, unregister+close, and check+open+read+close. 2. Read-only connections keyed by URI spelling (BLOCKING). SessionDB's read-only path opens file:/…/state.db?mode=ro; that string was fed to Path.resolve(), producing <cwd>/file:/…/state.db?mode=ro. No probe of the real Path could match, so read-only connections were invisible to the guard and their locks cancellable. Reproduced with no forced scheduling. Keys now come from PRAGMA database_list (canonical path), with an explicit tracking_path override. 3. Fail-open wrapper (HIGH). _connect_tracked_db() caught every exception and retried an untracked plain connect, so any error silently disabled the guard. Now only ImportError (scaffold installs without hermes_cli) falls back; real failures propagate. 4. Backup paths that warned and proceeded (MEDIUM). _backup_corrupt_db() and _backup_db_file() raw-read live databases; they now REFUSE when a connection is live rather than warning. Losing a forensic copy beats corrupting the database being rescued. Custom factories are no longer rejected (that broke legitimate callers) nor silently untracked -- the tracking close() is mixed into whatever factory is in play, including when an opener substitutes its own after the fact. WAL POLICY: #70055 is RESTORED, not reverted. My earlier justification was confounded -- the clean WAL result came from 3.53.1, which carries both the WAL-reset fix AND 3.51.0's broken-lock defenses, so it said nothing about the bundled 3.50.4. Re-measured on 3.50.4 with the lock fix in place: WAL 0/3 and DELETE 0/3, i.e. no evidence WAL is safer. Upstream still documents the WAL-reset bug through 3.51.2 as serious. Keeping new databases out of WAL until a fixed runtime ships is the conservative call, and the WAL policy does not belong in this root-cause fix. Six sabotage runs confirm each new test fails when its defect is reinstated (including two that initially did NOT -- the race test was rewritten to pause inside the byte read, and a separate test added for the opener-substituted factory path). 1124 targeted tests green.	2026-07-25 21:44:43 -07:00
teknium1	fbd5e5772b	fix(state): stop cancelling our own POSIX locks on live SQLite databases ... `hermes sessions optimize` could corrupt state.db. Root cause is Hermes, not the SQLite WAL-reset bug (#69784). close() on ANY file descriptor for a SQLite database cancels every POSIX advisory lock the process holds on that file, including a running VACUUM's EXCLUSIVE lock (sqlite.org/howtocorrupt.html section 2.2). Hermes byte-probed live databases in several hot paths: the zeroed-state.db detector runs on every SessionDB construction (and the gateway builds those constantly), and kanban's post-commit invariant check ran after every COMMIT. While VACUUM rewrote the file, those probes dropped its lock and let other processes write into it. A/B against the real code, only variable being the raw read: SQLite 3.50.4, VACUUM + concurrent writers, DELETE mode raw open/close during VACUUM 8 vacuums, 319 vacuum errors, 2/2 corrupt no raw read (control) 229 vacuums, 0 vacuum errors, 0/2 corrupt SQLite 3.53.1 (WAL-reset FIXED) reproduces identically: 2/2 corrupt. After this change: 0/4 corrupt, 0 vacuum errors. Because the upgraded runtime corrupts too, replacing the embedded SQLite does not fix this class; and because DELETE is where it reproduces, #70055's "force DELETE on vulnerable builds" mitigation steered users into the failing mode. That gate is reverted here: vulnerable builds get WAL again and still warn so operators can upgrade. - add hermes_cli/sqlite_safe_read.py: read page_count via PRAGMA over the existing connection instead of open()+seek(28); byte-level probes are restricted to before any connection exists and refused once one is live, with an explicit force= escape for offline artifacts (snapshots, archives) - track live connections in SessionDB and kanban's connect so that guard is enforced rather than merely documented - kanban's torn-extend check now only applies under a rollback journal; in WAL a committed page may still legitimately sit in the -wal file - revert the force-DELETE WAL gate and update the tests that pinned it Regression tests assert the behavioural contract (an external process stays locked out across Hermes' inspection calls) and were verified to fail when the old raw-open behaviour is restored.	2026-07-25 21:44:43 -07:00
Justin Bennington	bbee3011fd	test(runtime): cover managed SQLite cutover (E-949)	2026-07-24 16:00:03 -07:00
HexLab98	a94f8e69e1	test(state): cover SQLite WAL-reset version gate and doctor probe ... Assert the version matrix, fresh-DB DELETE fallback, already-WAL left alone (no checkpoint/DELETE), fixed-SQLite WAL path, and warn-only doctor output for vulnerable builds (#69784).	2026-07-23 17:32:38 +05:30