perf(state): keep delegate-child transcripts out of the trigram FTS index (schema v30)
On a fan-out-heavy install state.db reached 3.4 GB; 70% of message bytes
belonged to subagent sessions, and every one of those rows was also
indexed into messages_fts_trigram, whose shadow tables are ~2.6x the
text they cover (1,029 MB trigram vs 350 MB standard FTS on that DB).
session_search already hides source='subagent' sessions, so the
substring/CJK index bought nothing for them.
Extend the v29 cron exclusion: the messages_fts_trigram_src view, the
three sync triggers, and both deferred-backfill INSERT...SELECTs now use
one shared predicate (FTS_TRIGRAM_SESSION_SQL / fts_trigram_session_sql)
that skips sessions with source IN ('cron','subagent') or the
$._delegate_from creation marker (children spawned under a gateway turn
inherit the gateway's source). Compression/branch continuations carry
parent_session_id without the marker and stay indexed. Child rows remain
canonical in `messages` and fully indexed in the standard messages_fts
word index; explicit source_filter=['subagent'] CJK searches route to
LIKE like cron already did.
The v29 migration gate becomes `< 30` and reuses the same view-swap +
admitted rebuild, so existing installs purge historical child postings
once on open. Fresh DB with 2,000 x 2 KB child messages: 22.4 MB ->
12.5 MB (trigram shadow 10.09 MB -> 0.02 MB).
This commit is contained in:
+10
-3
@@ -24,7 +24,9 @@ from hermes_state_common import (
|
||||
FTS_STORAGE_VERSION,
|
||||
FTS_TOOL_CONTENT_PREFIX_CHARS,
|
||||
FTS_TOOL_FULL_CONTENT_HIGH_WATER_KEY,
|
||||
FTS_TRIGRAM_EXCLUDED_SOURCES,
|
||||
FTS_TRIGRAM_SQL,
|
||||
fts_trigram_session_sql,
|
||||
MAX_FTS5_QUERY_CHARS,
|
||||
SCHEMA_VERSION,
|
||||
_FTS_CJK_TRIGGERS,
|
||||
@@ -169,7 +171,7 @@ class SessionSearchMixin:
|
||||
"SELECT m.id, m.content, m.tool_name "
|
||||
"FROM messages m JOIN sessions s ON s.id = m.session_id "
|
||||
"WHERE m.id > ? AND m.id <= ? AND m.role <> 'tool' "
|
||||
"AND s.source <> 'cron' "
|
||||
f"AND {fts_trigram_session_sql('s')} "
|
||||
"AND NOT EXISTS (SELECT 1 FROM messages_fts_trigram_docsize d WHERE d.id = m.id)",
|
||||
(lo, hi),
|
||||
)
|
||||
@@ -329,7 +331,7 @@ class SessionSearchMixin:
|
||||
"SELECT m.id, m.content, m.tool_name "
|
||||
"FROM messages m JOIN sessions s ON s.id = m.session_id "
|
||||
"WHERE m.id > ? AND m.id <= ? AND m.role <> 'tool' "
|
||||
"AND s.source <> 'cron'",
|
||||
f"AND {fts_trigram_session_sql('s')}",
|
||||
(progress, upper),
|
||||
)
|
||||
# Publish progress in the same transaction as the rows it
|
||||
@@ -1922,7 +1924,12 @@ class SessionSearchMixin:
|
||||
# query explicitly filtering on role='tool' must therefore use
|
||||
# the LIKE fallback, which scans the base table directly.
|
||||
_wants_tool_rows = bool(role_filter) and "tool" in role_filter
|
||||
_wants_cron_rows = bool(source_filter) and "cron" in source_filter
|
||||
# Cron and subagent transcripts are excluded too (see
|
||||
# FTS_TRIGRAM_EXCLUDED_SOURCES); an explicit filter for them
|
||||
# must likewise scan the base table.
|
||||
_wants_cron_rows = bool(source_filter) and any(
|
||||
src in FTS_TRIGRAM_EXCLUDED_SOURCES for src in source_filter
|
||||
)
|
||||
|
||||
# ── CJK-bigram route (messages_fts_cjk, cjk_unicode61) ──────
|
||||
# When the bigram index is available it serves EVERY CJK query
|
||||
|
||||
Reference in New Issue
Block a user