fix(search): strip the FTS5 special characters the sanitizer was missing

_sanitize_fts5_query's strip step only removed +{}():"^ . Every other
character FTS5's grammar rejects outside a quoted phrase reached MATCH
raw and raised, and — as the step's own comment says about the colon it
was fixed for — the execute site swallows that into zero results. Session
search silently found nothing for ordinary queries:

  it's            fts5: syntax error near "'"
  gateway/run.py  fts5: syntax error near "/"
  user@host       fts5: syntax error near "@"
  a,b             fts5: syntax error near ","
  why?            fts5: syntax error near "?"
  e=mc2           fts5: syntax error near "="

Complete the class and assemble it with re.escape, because written as a
regex literal the backslash was eaten as an escape and never made it in
(C:\path\file still raised after the first pass).

Measured against a real FTS5 table over 651 realistic queries:
373 unparsable before, 77 after. The remainder is leading/trailing "." and
"-", which #43889 already covers.

% is deliberately left in: the CJK path falls back to a LIKE search that
needs it literal and escapes wildcards itself, so stripping it widened
those queries onto unrelated rows (test_cjk_like_escapes_wildcards).
This commit is contained in:
Drexuxux
2026-08-05 13:33:34 +03:00
committed by Teknium
parent 0569c001d0
commit c595dcb955
2 changed files with 86 additions and 1 deletions
+19 -1
View File
@@ -32,6 +32,18 @@ from hermes_state_common import (
# keep that logger identity so log filtering/capture behavior is unchanged.
logger = logging.getLogger("hermes_state")
# Characters FTS5's query grammar rejects outside a quoted phrase. Anything
# missing from this set reaches MATCH raw and raises, which the execute site
# swallows into zero results — the failure this strip step exists to prevent.
# Assembled through re.escape so the backslash cannot be eaten as a regex
# escape inside the class (it was, while the set was written as a literal).
#
# ``%`` is deliberately excluded: a CJK query falls back to a LIKE search that
# needs it preserved as a literal (that path escapes wildcards itself), so
# stripping it here widened those queries onto unrelated rows.
_FTS5_SPECIAL_CHARS = '+{}():"^@/#&|~[]<>,;!?$=\\\''
_FTS5_SPECIAL_RE = re.compile(f"[{re.escape(_FTS5_SPECIAL_CHARS)}]")
class SessionSearchMixin:
"""See module docstring — mixin for SessionDB (Search cluster)."""
@@ -1210,7 +1222,13 @@ class SessionSearchMixin:
# single ``content`` column, an unquoted colon query like ``TODO: fix``
# parses as ``column:term`` and raises "no such column" — swallowed at
# the execute site into zero results. Strip it like the others.
sanitized = re.sub(r'[+{}():\"^]', " ", sanitized)
# The class below is every character FTS5's query grammar rejects
# outside a quoted phrase. Anything omitted here reaches MATCH raw and
# raises, which the execute site swallows into zero results — the
# failure mode this step exists to prevent. Measured against a real
# FTS5 table: ``it's``, ``gateway/run.py``, ``user@host``, ``a,b`` and
# ``50%`` all raised before the class was completed.
sanitized = _FTS5_SPECIAL_RE.sub(" ", sanitized)
# Step 3: Collapse repeated * (e.g. "***") into a single one,
# and remove leading * (prefix-only needs at least one char before *)