Flip the state.db retention defaults per Teknium's decision on #54189:
- sessions.auto_prune: false -> true. A stock install now prunes ENDED
sessions inactive for retention_days at CLI/gateway/cron startup
(at most once per min_interval_hours). Open, pinned and mid-turn
sessions are never deleted; the only open rows touched are stale
automation sessions (#100903 sweep), which are closed, not deleted,
and aged a further full window before removal.
- sessions.retention_days stays 90 (already the default; verified).
- Auto-VACUUM is now additionally gated on the reclaimable fraction of
the file: PRAGMA freelist_count / page_count must exceed 25%
(AUTO_VACUUM_MIN_FREELIST_RATIO) on top of the existing
min_vacuum_interval_days throttle. Pruning a few small sessions on a
dense multi-GB DB no longer rewrites the whole file to reclaim a few MB.
Unknown ratio (pragma read failure) falls back to the time throttle.
Existing installs that explicitly set any sessions.* key keep their
values (load_config deep-merges DEFAULT_CONFIG under user YAML); only
unset keys pick up the new defaults. No _config_version bump needed.
cli-config.yaml.example documents the section commented-out so
installers that copy it verbatim never pin these as explicit settings.
Tests: ratio gate (below/above/at-threshold/unknown/override), real-DB
freelist ratio, default assertions, fresh-config startup hook reaches
the prune call, explicit opt-out respected, template-does-not-pin-keys.
The per-profile store partition (17ba992108, 5ffaed6e45, 5cc3da6827)
already keeps fresh rows apart, but legacy rows written to root state.db
before the partition still sat where the default profile's peer-tuple
fallback could adopt them: a Telegram DM's tuple (chat_id == user_id, no
thread) is identical for every bot. Three residual holes, closed with the
smallest predicate that fits main's design:
- hermes_state.find_latest_gateway_session_for_peer: the fallback query
now requires COALESCE(s.profile_name, <store owner>) = <store owner>
(owner via SessionDB._own_profile_name). Handles NULL legacy rows and
the single→multiplex migration case a key-namespace fence would break;
stores outside the profile tree (no derivable owner) are unchanged.
- gateway/session._recovered_row_allowed_for_active_profile: under
multiplexing no longer `return True` — the recovered row's agent:<ns>:
must match the REQUESTED key's namespace (the active profile is
meaningless when several profiles serve concurrently). Single-profile
behavior unchanged; keyless/unnamespaced rows stay adoptable.
- hermes_state create_session parent COALESCE: profile_name inherits only
when parent and child agree on agent:<ns>: (or either is keyless), so a
default child forked from a sibling row is not durably mislabelled.
Co-authored-by: pcaruba <31041167+pcaruba@users.noreply.github.com>
Co-authored-by: jiangtaoliu-source <308256854+jiangtaoliu-source@users.noreply.github.com>
Co-authored-by: 69k4xmdfm2-blip <275826864+69k4xmdfm2-blip@users.noreply.github.com>
Issue #76423: under multiplex_profiles a shared state.db keyed topic mode
and bindings only by Telegram chat_id/thread_id, so private-chat ids
collided across bots/profiles.
- Add profile_name to telegram_dm_topic_mode and telegram_dm_topic_bindings
- Schema v2→v3 rebuild; legacy rows migrate into the "default" namespace
- Keyword-only profile_name="default" on SessionDB topic APIs (compat)
Drop the no-field fallback vitest case and the get_compression_chain
unit test: the fallback is implied by the predicate (Boolean(undefined))
and the chain walk is covered by test_list_serves_full_lineage_ids_for_projected_rows
through the real list projection.
Compression rotates a conversation's tip id while tiles stay keyed by
whichever segment id they were opened with. focusOpenSession and
openSessionTile tested exact ids, so right after a rotation the same
chat read as 'not open' and opened again in a second tab — and a tile
keyed to a MIDDLE segment (the tip when it was opened) could no longer
prove it names the conversation at all, rendering as an untitled ghost.
The projected list row now carries the full chain
(SessionDB.get_compression_chain, served as _lineage_ids by
list_sessions_rich and the sidebar tree row), lineageAliases indexes
every segment, sessionMatchesStoredId accepts membership, and the tab
focus/open paths dedupe through the lineage instead of the exact id.
Older gateways omit the field and degrade to today's root/tip pairing.
Desktop's cold resume (defer_history + omit_messages, transcript paged over
REST) only ever holds the live tip segment in memory, but session.resume
bounded it against the FULL compression lineage (sessions.max_resume_messages,
default 20000). A Bot Chat with 85 compaction segments / ~29k lineage rows
behind a ~700-row tip was refused at 20001, sent zero model prompts, and sat on
"Waking up default…" forever — the healthiest possible session shape, rejected
by a guard sized for in-memory materialization.
- hermes_state: one `_resume_lineage_ids` definition shared by the resume
readers (get_resume_conversations, get_ancestor_display_prefix) and the
guard (assert_resume_safe / get_resume_message_count). Guard grows
`tip_only=` and names the scope it counted; the branch-aware lineage the
readers already used is now what the guard counts too (a /branch copy was
being counted against its parent's rows).
- tui_gateway session.resume: deferred, omit_messages and lazy resumes are
bounded by the tip; only the full in-memory lineage resume keeps the
lineage-wide bound. Deferred hydration falls back to tip-only history when
the lineage exceeds the limit instead of loading the rows the guard refused.
- CLI mid-setup tip-only path routes through the same guard instead of
borrowing assert_export_safe.
- docs: sessions.max_resume_messages / max_export_messages documented with the
per-surface scope.
Live repro (real SessionDB fixture, 85 segments / 29,226 lineage rows / 666 tip
rows, real tui_gateway.server.handle_request): before — deferred resume ->
4130; after — ok, hydrated history=666 prefix=0; the non-deferred full resume
still returns 4130 on the same fixture.
A concurrent WAL checkpoint / reset / frame-flush can surface SQLITE_IOERR
to a reader on a perfectly healthy database: a mode=ro connection cannot
perform the WAL recovery the read needs, because recovery writes the -shm
index and read-only mode refuses. The window is millisecond-scale.
Today that one-shot error escapes the SessionDB read-only constructor, and
GET /api/sessions turns it into a 500 the desktop reads as an authoritative
empty list.
Retry it, bounded, in the constructor so every read-only opener is covered —
the sidebar poll, cross-profile aggregation, recall, browse — rather than at
one route. A persistent IOERR still exhausts the budget and propagates.
Remaining transient failures answer 503, so the client keeps the list it has.
On the write path, BEGIN IMMEDIATE can hit the same transient IOERR before
the callback runs. That one is safe to retry on the same connection because
nothing has been mutated; once the callback starts, settlement is unknown and
the error propagates. Never close()+reopen to heal it — close() cancels this
process's POSIX advisory locks on the file for every sibling connection, and
a list poll's reader must stay disposable so a replaced state.db is observed
and the pre-repair forensic backup stays reachable.
Fixes#100436
Co-authored-by: rkfshakti <rkfshakti@users.noreply.github.com>
Co-authored-by: AKAZIK-py <AKAZIK-py@users.noreply.github.com>
`_init_schema` decided whether the FTS triggers needed repair by comparing
the live trigger count against `len(_FTS_TRIGGERS)`, the full six-name set.
Three of those six are the `messages_fts_trigram_*` triggers, and they are
declared only inside `FTS_TRIGRAM_SQL` / `LEGACY_FTS_TRIGRAM_SQL`, whose
`CREATE VIRTUAL TABLE ... tokenize='trigram'` needs a tokenizer SQLite only
gained in 3.34.
On an older build `_ensure_fts_schema` soft-fails that DDL by design (via
`_is_trigram_unavailable_error`) and returns False, so those three triggers
can never be created. The count is therefore pinned at 3, `3 < 6` is
permanently true, and the repair path ran on every single `SessionDB` open,
forever, while holding the SQLite write lock. It never converged: every
`hermes` command, gateway start, dashboard request and cron tick paid a full
re-index of the message corpus. That is ordinary LTS territory — Ubuntu
20.04 ships 3.31, RHEL/CentOS 8 and Alibaba Cloud Linux ship 3.26, and
Hermes has no minimum-SQLite gate precisely because it is supposed to
degrade gracefully here.
The v23 repair also ends by clearing `fts_rebuild_high_water` and
`fts_rebuild_progress`, which is correct after a genuine full rebuild but
means an interrupted `hermes sessions optimize-storage` silently lost its
resume point on the next open, restarting the chunked backfill from zero
every time.
Fix: keep `_FTS_TRIGGERS` as the single source of truth and derive two
subsets from it, then measure each half against the DDL that can actually
create it. `_fts_trigger_count` takes an optional `names` sequence
(defaulting to the full set, so no caller changes), and both branches gate
on `base_triggers_missing or (trigram_enabled and trigram_triggers_missing)`.
The counts are still taken before the DDL runs so they describe the
pre-repair state, while `trigram_enabled` is only known afterwards — hence
the combination at the `if` rather than at the assignment.
Behaviour is unchanged wherever the tokenizer exists: a genuinely missing
trigram trigger on a capable host still triggers the rebuild. Only the
permanently unsatisfiable comparison changes.
Rework of salvaged PR #6372 (@ag9920) onto current main:
- /save promoted from CLI-only JSON snapshot to a cross-platform session
export: `/save [json|md|html] [filename] [redact]` on CLI and every
gateway platform (sent as a document via adapter.send_document).
- Rendering routes through the existing shared renderers
(hermes_cli/session_export.py + session_export_html.py) instead of the
PR's new hermes_state formatter — new helpers normalize_save_format /
render_session_for_save / default_save_filename are shared by both
surfaces.
- `redact` arg runs the export through the force-mode secret redaction
pass (session_export_md.redact_session_data) before writing.
- Gateway handler awaits AsyncSessionDB correctly, sanitizes user-supplied
filenames with basename, and lands in gateway/slash_commands.py (the
handlers moved out of gateway/run.py since the PR was authored).
- /export stays profile export (name collision resolved: session export
lives on /save).
- Slack 50-slash cap curation: /platform moves to the /hermes-only set to
free a native slot for /save (parity test updated rationale comment).
- Folds in PR #62268 (@briandevans): None title/model coalescing in the
single-session HTML export.
Closes#4249. Closes#51200.
Every search route in _search_messages_impl (FTS, CJK bigram, trigram,
LIKE fallback, rebuild-gap supplement) selected m.content, then the
result tail popped it unread. On DBs with multi-MB tool rows, each
search read and materialized up to `limit` full rows only to discard
them. Snippets come from snippet()/substr() in SQL and the context
window is re-fetched by id, so no code path ever read the column.
Drop the column from all six SELECT lists. Returned dicts are
unchanged: content was never part of the public result (the pop ran
before return), and tests/test_hermes_state.py already documents that
contract.
(cherry picked from commit d0c3af167e7dd4eb18e1bea29ba107a94911ea24)
- tests/cron/test_sessiondb_init_hang.py: add threading/time imports the
salvaged late-close regression tests rely on.
- tests/test_hermes_state.py: drop
test_close_closes_wal_read_connection_created_on_worker_thread — main
replaced per-thread WAL reader ownership with the pooled read-connection
design (permits + checkout/return), so cross-thread reader draining no
longer exists in the form the test asserted.
SessionDB could leave native SQLite handles open when construction failed
partway through schema/pragma/FTS/repair/lock/interrupt handling. Other
short-lived callers (MCP reads/polling, session search, reactions, trace
upload, insights, shutdown recovery) opened temporary SessionDB handles
without a complete ownership boundary. API-server profile caches and
RetainDB shutdown had similar late-close races. Under sustained load this
exhausted file descriptors (EMFILE).
- Close partially initialized SessionDB connections on every constructor
exception path via a finally block guarded by an initialization-complete
flag.
- Close temporary/cross-profile SessionDB handles in finally blocks across
CLI, MCP, search, trace, reactions, insights, and recovery paths.
- Add API-server per-profile cache ownership and disconnect cleanup.
- Make RetainDB writer-queue shutdown exception-safe: track connections per
thread, close on worker exit, reject new enqueues after shutdown starts,
and sweep any connections left by short-lived threads.
- Add regression coverage for constructor failures, worker-thread readers,
API disconnect failures, shutdown recovery, RetainDB late enqueue, and
foreign-loop async clients.
Salvage notes: the original PR's per-thread WAL-reader ownership changes
were superseded by main's read-connection pool (permits + checkout/return);
its cron timeout-abandon fix is credited separately to #72822's earlier
identical fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
After `hermes update`, an existing state.db on an old schema made every
GET /api/sessions poll fail with sqlite3.OperationalError "no such
column: s.last_read_at" (or s.last_activity_at) until something
unrelated forced a writable open — the desktop sidebar showed "No
sessions yet" while every row sat intact on disk (#79531, #80037).
Two remaining root causes (the stale hand-written read probe was
already replaced by the SCHEMA_SQL-derived probe on main, prototyped in
draft PR #80030 by @Tilly-YL):
1. Migrations ran lazily: _init_schema/_reconcile_columns only ran on a
writable open, typically the user's first NEW session. The dashboard
backend now schedules one writable open of its own state.db from the
lifespan (daemon thread, never blocks the ready-probe socket, never
raises), so the store is brought current before the first session-
list poll on every `hermes serve` / `hermes dashboard` / Desktop
headless entrypoint.
2. _reconcile_columns caught sqlite3.OperationalError around every
ALTER TABLE ADD COLUMN and logged at DEBUG. Lock contention from
orphaned sibling backends made the ALTER fail silently — startup
"succeeded" with a half-reconciled schema, and the open-time lock
patience (#74478) never saw the error because it was swallowed
inside first. Now: "duplicate column" races stay at DEBUG,
locked/busy re-raises so _connect_and_init_with_lock_patience
retries the whole idempotent init with jittered backoff, and any
other failure (e.g. un-ADDable NOT NULL) logs at WARNING.
Regression tests: a store missing sessions.last_read_at is healed by
the eager startup reconcile and serves list_sessions_rich; a locked
ALTER propagates and is retried to success by the open lock patience;
duplicate-column races stay quiet; other ALTER failures warn.
Fixes#79531Fixes#80037
Reported-by: @yenhunghuang (#79531) and @FLOW3R0111 (#80037)
Root-cause analysis: @wangyi0177-eng (stale read probe) and
@www654cc-pixel (_reconcile_columns DEBUG-swallow under lock
contention); draft PR #80030 by @Tilly-YL prototyped the probe fix.
Follow-ups to the salvaged #84009 commits:
- Add 'session_switch' to _RESET_END_REASONS: a reset continuation's
parent can be promoted to session_switch (resume the reset parent,
then switch away), which permanently hid pre-marker legacy children —
reopen-time stamping cannot rescue them because the parent is being
ended, not reopened. Probe-verified before/after.
- Share the legacy reset-child heuristic via _legacy_reset_child_sql()
so _RESET_CHILD_SQL and reopen_session()'s stamping UPDATE cannot
drift, and derive find_latest_gateway_session_for_peer's two recovery
fence literals from _RESET_END_REASONS_SQL (was a third hand-written
copy of the same set).
- Exclude reset children (marker + legacy shape) from the
resolve_resume_session_id forward walker: resuming a reset parent
could redirect into the post-reset conversation — the exact context
the user reset away. Regression tests cover both shapes plus the
walker's original compression-tip behavior; mutation-checked.
`sessions optimize` could consume several GB of disk instead of freeing
any, filling the host to 100% on exactly the large databases it exists to
shrink.
Two causes, both in the WAL lifecycle:
1. No `journal_size_limit`. SQLite defaults to -1 (unlimited), so after a
checkpoint the WAL is reused in place and never truncated —
`state.db-wal` permanently keeps the high-water mark of the largest
transaction ever run. `hermes_cli/kanban_db.py` already bounds its WAL
with `wal_autocheckpoint=100`; the session store, by far the larger
database, had no equivalent.
2. `vacuum()` checkpoints BEFORE `VACUUM` but not after. VACUUM rewrites
every page through the WAL, so the pre-checkpoint does nothing about
the slack VACUUM itself creates.
Measured on a 3.0 GB state.db: `hermes sessions optimize` reported
"3143.9 MB -> 3155.1 MB (reclaimed -11.2 MB)" while leaving a 3.07 GB
state.db-wal behind. Free space fell from 6.9 GB to 772 MB (100% full)
and stayed there. A manual `PRAGMA wal_checkpoint(TRUNCATE)` recovered
the full 3.07 GB, confirming it was slack, not data.
Fix: set `journal_size_limit` (64 MiB) when enabling WAL, and truncate
the WAL again after VACUUM. Both are best-effort and never raise — a
failure costs disk slack and must not stop the DB from opening.
Tests assert the contract (limit is a finite positive bound; VACUUM does
not leave an oversized WAL) rather than pinning the byte count, which is
a tunable. They skip where WAL is unavailable — including hosts where
Hermes falls back to journal_mode=DELETE due to the SQLite 3.50.4
WAL-reset bug.
Verified: 462 passed / 3 skipped in tests/test_hermes_state.py, and
_apply_wal_size_limit flips a real WAL database from -1 to 67108864.
Tested on Linux (aarch64, Python 3.11).
SessionDB.close() ran `PRAGMA wal_checkpoint(TRUNCATE)`. Every cron
run_agent opens and closes its own transient SessionDB, so on a busy
fleet this fired a full WAL reset many times an hour, racing the
gateway's long-lived writer on a large WAL database and tearing hot
B-tree pages -- structurally the same corruption this module's own
periodic checkpoint was already switched to PASSIVE to avoid (#45383).
Only close() and two manual-maintenance paths still used TRUNCATE.
Route every checkpoint on the shared state.db through PASSIVE:
- close() (hermes_state.py)
- pre-VACUUM in vacuum() (hermes_state.py)
- post-optimize-storage (hermes_state_search.py)
PASSIVE never resets/truncates the WAL and never takes the exclusive
checkpoint lock, so it cannot lose a transient closer's race with the
live writer. The WAL is instead bounded by `journal_size_limit` and the
writer's natural post-checkpoint reset. TRUNCATE belongs only on a
sole-opener/quiescent connection (e.g. offline maintenance); this change
does not try to detect that -- PASSIVE is the safe default.
Diagnosed as the root cause of three state.db B-tree corruptions in
2026-08: damage localized to the hottest-written pages (gateway_routing
and the sessions indexes), with whole zero-filled pages still live and
off the freelist -- the checkpoint/reset-race signature, not disk or
application SQL.
Tests: tests/test_wal_checkpoint_strategy.py now asserts PASSIVE at
close(), before vacuum(), and after optimize_fts_storage() VACUUM;
tests/test_hermes_state.py asserts close() likewise. Focused run:
226 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
find_latest_gateway_session_for_peer filtered non-recoverable rows out of
candidacy BEFORE ordering, so recovery could search behind a /new reset
boundary and resurrect an older still-open row for the same peer —
silently restoring the exact context the user reset.
Rebuilt against the #82633 finder (has-messages ranking +
COALESCE(last_activity_at, started_at) recency): the fence is expressed
as a NOT EXISTS guard inside both the exact-key and peer-fallback
queries — a candidate is rejected when an intentional boundary row
(session_reset / session_switch / idle / daily / suspended /
resume_pending_expired) for the same peer ended after the candidate's
last activity. If the conversation's most recent event is an intentional
reset, recovery returns nothing rather than reaching behind it.
Cherry-picked from #68617 and adapted to the rewritten finder.
(cherry picked from commit bb2c562a165d91e00f64d42cf7495e6c8a5da9d7)
Closes the residual the contributor's own triage comment flagged: % was
excluded from the special-char class to protect the CJK LIKE fallback,
but a non-CJK query never reaches that fallback (is_cjk gates it), so
'50%' still hit MATCH raw and silently returned zero results. Strip %
whenever the sanitized query contains no CJK; the CJK path keeps its
pre-existing contract. Regression tests for both directions.
_sanitize_fts5_query's strip step only removed +{}():"^ . Every other
character FTS5's grammar rejects outside a quoted phrase reached MATCH
raw and raised, and — as the step's own comment says about the colon it
was fixed for — the execute site swallows that into zero results. Session
search silently found nothing for ordinary queries:
it's fts5: syntax error near "'"
gateway/run.py fts5: syntax error near "/"
user@host fts5: syntax error near "@"
a,b fts5: syntax error near ","
why? fts5: syntax error near "?"
e=mc2 fts5: syntax error near "="
Complete the class and assemble it with re.escape, because written as a
regex literal the backslash was eaten as an escape and never made it in
(C:\path\file still raised after the first pass).
Measured against a real FTS5 table over 651 realistic queries:
373 unparsable before, 77 after. The remainder is leading/trailing "." and
"-", which #43889 already covers.
% is deliberately left in: the CJK path falls back to a LIKE search that
needs it literal and escapes wildcards itself, so stripping it widened
those queries onto unrelated rows (test_cjk_like_escapes_wildcards).
With sessions.max_*_messages: 0 the guards previously still ran an
unbounded COUNT (full lineage for resume) — the exact pathological work
disabling them is meant to avoid. Live callers use the raise side
effect only, so return 0 without touching the messages table.
OFFSET paging made the streaming export O(n^2) on huge transcripts;
after_id keyset paging keeps each page seek O(1). Adds after_id to
SessionDB.get_messages (ascending-only, guarded against latest/offset
combos).
sessions.max_resume_messages / sessions.max_export_messages (default
20000, 0 disables) replace the hardcoded hard-rejects, and the CLI
'sessions export' guard becomes per-session instead of cumulative so
full-DB backups of many small sessions keep working. Error guidance now
points at the config override instead of the (corruption-only) repair
command.
_fts_teardown_trash_step deleted rows via 'WHERE key IN (SELECT key
LIMIT N)' — each chunk's subquery re-scanned from the start of the
table, so chunk k skipped past (k-1)xN already-deleted rows: O(n²)
total row visits. On a v22 shadow table with ~230K rows that is on the
order of 10^8 row visits, turning optimize-storage teardown into a
multi-hour grind on slow disks, with a write lock held per chunk.
Single-column INTEGER-PK trash tables now drain via a fts_teardown_<tbl>_progress
high-water marker mirroring fts_rebuild_step: each chunk claims rows
past the marker (SELECT ... WHERE key > ? ORDER BY key LIMIT N), deletes
the claimed range, and publishes the new marker in the same transaction.
Per-chunk work is bounded → O(n) total.
TEXT-PK tables (the FTS config shadow table, pk like 'version') and
compound-key tables fall back to the legacy chunked delete — those are
small by construction.
Fixes#79324
test_hermes_state.py was the slowest file in the suite (46.7s in CI's
durations cache) and therefore the LPT floor: no test slice can finish
faster than its slowest file, which caps how far slicing the test matrix
wider can cut the merge-gate critical path.
Profiling (cProfile on the slowest tests) found the time was dead, not
work:
1. time.sleep in optimize_fts_storage's inter-chunk throttle — 4.1s of
a 4.6s migration test. The throttle exists so a LIVE gateway/CLI
sharing the DB isn't starved of the write lock; tests run against a
private tmp-path DB with no concurrent process, so the sleep protects
nobody. New autouse fixture zeroes _FTS_REBUILD_MIN_PAUSE /
_FTS_REBUILD_DUTY_FACTOR for this file (~20s saved). No test asserts
on wall-clock pacing, so nothing weakens.
2. TestGetMessagesPagination._seed appending 3000 messages one
append_message (= one commit, and off WAL one fsync) at a time —
~10s of seeding before the query under test even ran. Switched to
append_messages_batch (one write transaction), the API the docstring
of which exists for exactly this shape. The perf contract the seed
feeds still discriminates: measured 11 progress-handler steps on the
indexed path vs 855 on the forced scan path, against the unchanged
300 threshold.
Measured (local, 3 runs + canonical runner):
before: 187 passed in 51.8s
after: 187 passed in 9.1-16.1s (canonical scripts/run_tests.sh: 14.9s)
Zero production code touched; 187 tests before and after.
The quadruple-backslash pattern arm is the trickiest byte sequence in the
fix and had no direct coverage — 'simplifying' it to a double backslash
would break Windows child matching with every test still green. Mutation
checked: weakening the arm fails this test.
_cwd_prefix_clause builds "cwd is this directory or under it" for session
listing, workspace resume and prune/archive. The two LIKE arms bound the
raw prefix, so `_` and `%` acted as wildcards on a value that is a path:
cwd_prefix="/home/me/my_project"
main -> ['sibling', 'target'] # /home/me/myXproject/src matched too
fix -> ['target']
`_` matches any single character, so a same-length sibling directory with
children falls inside the pattern. prune_sessions() deletes the rows it
matches (and their on-disk transcripts), so an unrelated project's history
goes with it.
Escape the needle and pair both arms with ESCAPE, the convention the rest
of this file already uses; the literal separator backslash in the Windows
pattern is escaped for the same reason. The `=` arm is an exact compare and
keeps the raw prefix, so directory-and-children matching is unchanged.
Follow-up to the *_like filters in #78681, kept separate because this helper
is shared by four call sites beyond prune.
_prune_filter_where documents title_like / model_like / branch_like as
"case-insensitive substring matches", and the CLI confirmation renders them
as "title contains 'X'". They were bound straight into a bare LIKE, so `_`
matched any single character and `%` any run.
The builder backs prune_sessions(), which deletes session rows and their
on-disk transcripts, so the over-match is unrecoverable: pruning
title_like="user_auth" also destroys "user-auth", "userXauth" and
"user auth". `_` is not exotic here -- git branch names and session titles
carry it routinely.
Escape the operator's needle and add ESCAPE '\' to the three clauses, the
same convention the rest of this file already uses for LIKE queries. Match
direction is unchanged for needles without wildcards.
Left alone: _cwd_prefix_clause has the same unescaped shape but is shared
by four call sites beyond prune, so it is a separate change.
E2E guard for the salvaged PR #71755: database.cache_size/mmap_size/
temp_store from config.yaml must reach the writer connection, the
read-only cross-profile attach, and the WAL per-thread reader — and a
default install (no database: keys) must keep byte-identical SQLite
defaults on every connection type. Also covers integer-coercion
rejection of garbage values for the three new keys.
cache_size uses -16000 (not the doc example -2000) because -2000 is
SQLite's compiled-in default and would not discriminate a regression.
```
Error: optimization failed: no such table: messages_fts_trigram
No data was lost. Re-run to resume.
```
on any install where the trigram FTS index is legitimately absent. The failure is
deterministic — re-running can never make progress, because the crash happens at the same
point every time — so the database is permanently stuck on the legacy high-footprint FTS
layout with no supported way forward.
Observed on a 5.4 GB production `state.db`. After the fix the same database optimized
successfully and shrank to 3.3 GB.
The trigram index is absent whenever the runtime cannot maintain it. On a SQLite build
without the `trigram` tokenizer, `_ensure_fts_schema()` returns `False`, so `__init__`
leaves `self._trigram_available = False` and no `messages_fts_trigram` table on disk. This
is a **supported degraded runtime**, not damage — CJK/substring search falls back to
`LIKE` and everything else works normally. `_is_fts5_unavailable_error()` and
`_warn_trigram_unavailable()` exist specifically to make this path graceful.
Two code paths write the boundary sweep for the deferred FTS rebuild, and only one of them
respects that flag:
| Function | Trigram `INSERT` guarded? |
|---|---|
| `fts_rebuild_step()` | ✅ `if include_trigram:` where `include_trigram = self._trigram_available` |
| `_fts_rebuild_finish()` | ❌ unconditional |
`_fts_rebuild_finish()` runs the boundary sweep at the *end* of the backfill. Its
unguarded `INSERT INTO messages_fts_trigram …` raises `OperationalError`, which propagates
out of `optimize_fts_storage()` and aborts the entire optimization — *after* the backfill
has already completed. Hence the characteristic output showing 100% progress immediately
before the error:
```
Rebuilding index: 100% (909,671/909,671)
Error: optimization failed: no such table: messages_fts_trigram
```
There is a second, quieter consequence. The teardown phase that reclaims the demoted
`fts_v22_trash_*` shadow tables runs *after* the backfill phase in
`optimize_fts_storage()`. Because the crash happens before teardown is ever reached, those
tables are never emptied or dropped — so the space the migration was supposed to reclaim
stays allocated indefinitely, and the leftover trash tables look (misleadingly) like
evidence of a half-finished migration.
Build a populated v23 database, set the deferred-rebuild markers, then reopen it on a
runtime where `_ensure_fts_schema('messages_fts_trigram', …)` returns `False` (exactly
what a SQLite build without the trigram tokenizer produces) and call
`optimize_fts_storage()`:
```
[precondition] trigram absent, _trigram_available=False, rebuild pending ✓
RED ✗ optimize_fts_storage raised OperationalError: no such table: messages_fts_trigram
```
With this patch applied, unchanged harness:
```
optimize_fts_storage returned {'ok': True, 'vacuumed': None}
GREEN ✓ optimize ok; markers cleared; base FTS 'zebra' -> 200 hits
```
Full harness and transcripts in `TEST-EVIDENCE.md`.
Gate the sweep on `self._trigram_available`, exactly as `fts_rebuild_step()` already does:
```python
include_trigram = self._trigram_available
def _do(conn):
...
if include_trigram:
conn.execute("INSERT INTO messages_fts_trigram(...) ...")
```
The base `messages_fts` sweep and the marker cleanup are untouched, so the rebuild still
finalizes correctly and the index remains complete for every row it is responsible for.
The fix does not disable or weaken search to dodge the error — the regression tests assert
that base FTS still returns results afterwards.
`TestFtsRebuildFinishWithoutTrigram` in `tests/test_hermes_state.py`:
- `test_rebuild_finish_skips_trigram_when_unavailable` — drives `_fts_rebuild_finish()`
directly on a trigram-less runtime; asserts it completes, clears both rebuild markers,
and leaves base FTS searchable.
- `test_optimize_fts_storage_succeeds_without_trigram` — end-to-end through the public
`optimize_fts_storage()` entry point; asserts `ok=True`, markers cleared, search intact.
Both use the existing `_NoTrigramConnection` helper already in the file. Both fail on
`main` with `no such table: messages_fts_trigram` and pass with this patch.
`tests/test_hermes_state.py` passes in full (463 tests → 465 with these two). `ruff` clean.
This PR is the crash only.
A companion PR narrows `_db_opens_cleanly()` so that
`hermes sessions repair --check-only` stops reporting a write-broken FTS schema as
healthy — the gap that makes this class of problem hard to diagnose in the first place.
The two are independent and can land in either order.
Adds two regression tests for the #59077 batch: (1) _get_session_rich_rows_batch(compact_rows=True) uses the schema-derived compact projection (no system_prompt, git_branch/git_repo_root kept); (2) list_sessions_rich(compact_rows=True) threads compact_rows through the compression-tip projection call site. Mutation-checked: hardcoding compact_rows=False at the call site fails test 2.
list_sessions_rich()'s compression-root projection called
_get_session_rich_row() once per root — a separate single-row query per
compression root on every session-list render. Resolve every tip id
first, then fetch all tip rows in one WHERE id IN (...) query via the
new _get_session_rich_rows_batch().
_get_session_rich_row() is now a thin wrapper over the batch method, so
the enriched SELECT (preview + last_active) lives in exactly one place —
future column changes (e.g. #42196's include_system_prompt) only touch
one query.
get_compression_tip()'s chain walk is untouched; it's a genuine
per-session graph walk with branch/delegate-exclusion and race handling,
and batching it safely is out of scope here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-up (#67341): on a freshly initialized state.db (before
ANALYZE has run) the source-filtered branches of _get_tool_usage /
_get_skill_usage did not select idx_messages_assistant_calls_by_session
— the optimizer drove from idx_sessions_source_id and probed each
session's messages via idx_messages_session_active, scanning non
tool-call rows. Pin the index with INDEXED BY on all four fixed-predicate
branches so the plan is deterministic for both the unfiltered and
source-filtered scopes without depending on statistics.
Safe because the index is declared in SCHEMA_SQL (created by every
read-write SessionDB._init_schema) and every InsightsEngine caller opens
a read-write SessionDB; read-only attachments (which skip schema init)
are never used for insights.
Extract the four queries into class constants and add tests: query-plan
coverage for both scopes without ANALYZE, row-level equivalence between
pinned and un-pinned forms, and an assertion that INDEXED BY fails loudly
if the index is absent.
InsightsEngine._get_tool_usage and _get_skill_usage scan messages for
role='assistant' AND tool_calls IS NOT NULL, but no index aligns with
that predicate, so SQLite scans the full messages table on a large
state.db. Add a partial index over exactly those rows.
role and tool_calls are base columns in the messages table, so the index
lives in SCHEMA_SQL (created on both fresh and existing databases via the
executescript on every open) rather than DEFERRED_INDEX_SQL.
Adds schema regression coverage (fresh + reopened DB, plan uses the index)
and an Insights regression test proving tool/skill output is identical with
and without the index present.
Fixes#67341
Two places were using SELECT COUNT(*) when they only needed a boolean:
- has_any_sessions() called session_count() > 1 (full table scan)
- delete_session() used SELECT COUNT(*) WHERE id=? (full matching scan)
Fix:
- Add session_count_ge(n) to SessionDB — short-circuits via
SELECT 1 FROM sessions LIMIT n, returns bool
- has_any_sessions() uses session_count_ge(2) instead of session_count() > 1
- delete_session() uses SELECT 1 ... LIMIT 1 with fetchone() is None
- Add tests for session_count_ge
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
Demote wrote the empty v23 schema via executescript inside BEGIN IMMEDIATE,
which commits early and can leave trash + empty indexes without rebuild
markers. Re-run then tore down trash and stamped fts_storage_version with
docsize=0, permanently losing historical session search.
Stage markers with the demote, create schema only after they are durable,
heal empty-index bookkeeping on resume, and refuse settle until the base
index is populated. Settle refusal returns ok=False instead of raising,
and resume fails fast if the base v23 table cannot be re-created.
Orphan-marker repair only resets a missing fts_rebuild_progress to 0 once
the index is known empty: the chunk worker replays its whole selected id
range without an anti-join, so a partially indexed DB that lost only its
progress key is first reset to a known-empty surface, then rebuilt.
Ported onto the SessionDB mixin split (hermes_state_search.py /
hermes_state_schema.py).
The RO branch's new FTS capability probe raises sqlite3.DatabaseError on
a malformed store (the probe itself only catches OperationalError). The
outer __init__ handler re-raises without closing self._conn, leaking a
tracked connection for the process lifetime — which makes
_backup_db_file refuse its raw-copy, so the writable heal that follows
(web_server's stale-schema/malformed reopen) repairs the store WITHOUT
the forensic backup repair_state_db_schema promises. Close-then-reraise
on any probe failure, mirroring _open_probed's cleanup discipline.
Regression test: corrupt sqlite_master (duplicate messages_fts row),
assert the failed RO open leaves no live tracked connection and the
subsequent writable heal creates its malformed-backup file. Mutation-
checked: no-oping the cleanup handler makes the test fail.
Every ORDER BY id query on the messages table sorted or scanned the
whole session: get_messages_around's window seek, latest_message_row_id
(LIMIT 1), and get_messages' full-load ordering all paid O(session
history) per call — hot mid-turn via session_search and reactions.
messages.id is an original column (INTEGER PRIMARY KEY AUTOINCREMENT),
so the index lives in SCHEMA_SQL next to idx_messages_session — no
legacy-column migration hazard (the kanban lesson from #28776 does not
apply).
Measured (real schema, one 20k-message session, median of 30):
get_messages_around 7.08 -> 0.22 ms (32x), latest_message_row_id
3.37 -> 0.011 ms (307x), get_messages full load 111.6 -> 98.6 ms
(1.13x — remaining cost is row deserialization, not the sort).
Window results byte-identical at probe points across the session.
Tests: VM-step pin (get_messages_around bounded work, calibrated
~12 vs ~855 handler calls, threshold 300 — fails without the index)
and window parity with/without the index. No EXPLAIN/plan text
(behavior contracts, AGENTS.md).
Teknium