fix(search): strip the FTS5 special characters the sanitizer was missing
_sanitize_fts5_query's strip step only removed +{}():"^ . Every other
character FTS5's grammar rejects outside a quoted phrase reached MATCH
raw and raised, and — as the step's own comment says about the colon it
was fixed for — the execute site swallows that into zero results. Session
search silently found nothing for ordinary queries:
it's fts5: syntax error near "'"
gateway/run.py fts5: syntax error near "/"
user@host fts5: syntax error near "@"
a,b fts5: syntax error near ","
why? fts5: syntax error near "?"
e=mc2 fts5: syntax error near "="
Complete the class and assemble it with re.escape, because written as a
regex literal the backslash was eaten as an escape and never made it in
(C:\path\file still raised after the first pass).
Measured against a real FTS5 table over 651 realistic queries:
373 unparsable before, 77 after. The remainder is leading/trailing "." and
"-", which #43889 already covers.
% is deliberately left in: the CJK path falls back to a LIKE search that
needs it literal and escapes wildcards itself, so stripping it widened
those queries onto unrelated rows (test_cjk_like_escapes_wildcards).
This commit is contained in:
+19
-1
@@ -32,6 +32,18 @@ from hermes_state_common import (
|
||||
# keep that logger identity so log filtering/capture behavior is unchanged.
|
||||
logger = logging.getLogger("hermes_state")
|
||||
|
||||
# Characters FTS5's query grammar rejects outside a quoted phrase. Anything
|
||||
# missing from this set reaches MATCH raw and raises, which the execute site
|
||||
# swallows into zero results — the failure this strip step exists to prevent.
|
||||
# Assembled through re.escape so the backslash cannot be eaten as a regex
|
||||
# escape inside the class (it was, while the set was written as a literal).
|
||||
#
|
||||
# ``%`` is deliberately excluded: a CJK query falls back to a LIKE search that
|
||||
# needs it preserved as a literal (that path escapes wildcards itself), so
|
||||
# stripping it here widened those queries onto unrelated rows.
|
||||
_FTS5_SPECIAL_CHARS = '+{}():"^@/#&|~[]<>,;!?$=\\\''
|
||||
_FTS5_SPECIAL_RE = re.compile(f"[{re.escape(_FTS5_SPECIAL_CHARS)}]")
|
||||
|
||||
|
||||
class SessionSearchMixin:
|
||||
"""See module docstring — mixin for SessionDB (Search cluster)."""
|
||||
@@ -1210,7 +1222,13 @@ class SessionSearchMixin:
|
||||
# single ``content`` column, an unquoted colon query like ``TODO: fix``
|
||||
# parses as ``column:term`` and raises "no such column" — swallowed at
|
||||
# the execute site into zero results. Strip it like the others.
|
||||
sanitized = re.sub(r'[+{}():\"^]', " ", sanitized)
|
||||
# The class below is every character FTS5's query grammar rejects
|
||||
# outside a quoted phrase. Anything omitted here reaches MATCH raw and
|
||||
# raises, which the execute site swallows into zero results — the
|
||||
# failure mode this step exists to prevent. Measured against a real
|
||||
# FTS5 table: ``it's``, ``gateway/run.py``, ``user@host``, ``a,b`` and
|
||||
# ``50%`` all raised before the class was completed.
|
||||
sanitized = _FTS5_SPECIAL_RE.sub(" ", sanitized)
|
||||
|
||||
# Step 3: Collapse repeated * (e.g. "***") into a single one,
|
||||
# and remove leading * (prefix-only needs at least one char before *)
|
||||
|
||||
Reference in New Issue
Block a user