The zero-result fallback prefers messages_fts_cjk when built: exact
ranked token match for Latin runs unicode61 fused onto CJK, including
<3-char tokens the trigram leg can't recover.
Integration layer for the cjk_unicode61 tokenizer, rebuilt on the v23
schema (the contributed integration in PR #65544 predated it):
- messages_fts_cjk: external-content FTS5 over a tool-row-excluding view
(same v23 storage discipline as the trigram index it supersedes — zero
inline text copies). Serves EVERY CJK query shape the legacy routing
split between trigram (>=3 chars/token) and LIKE full scans (1-2 char
tokens). Lone 1-char CJK runs and role_filter=['tool'] queries keep
their legacy routes.
- Dedicated marker pair (fts_cjk_rebuild_high_water/progress) gates the
id-scoped triggers, so a cjk-only backfill never gates the complete
messages_fts/trigram triggers.
- Transitions ride (the existing
throttled/resumable chunk engine): fresh DBs are born with the index;
legacy v22 DBs land on v23+cjk in one run; already-optimized v23 DBs
gaining the tokenizer get a marker-gated backfill; live writes are
indexed immediately in every case.
- Tokenizer-loss self-heal: a process that can't load the extension drops
the cjk triggers (writes keep working), leaves a stale breadcrumb, and
the index is rebuilt from scratch on the next optimize run — triggers
are never reinstalled over a gap (external-content 'delete' on an
unindexed rowid is the FTS5 corruption hazard the marker gating exists
to prevent).
- Capability classification: 'no such tokenizer: cjk_unicode61' joins the
degraded-runtime error class everywhere (read probe, write probe,
repair) so tokenizer absence is never misclassified as corruption.
- Config: sessions.cjk_fts (default on, inert without the .so) and
sessions.search_slow_ms in config.yaml, bridged to env by CLI + gateway
(startup + per-turn reload). build.sh falls back to vendored SQLite
headers so no libsqlite3-dev is needed.
Slow-query log path attribution updated: fts_cjk / fts5 / trigram /
like_scan. Tests: 14 lifecycle tests (fresh/legacy/stale/backfill paths,
tokenizer-loss round-trip) + 5 config-bridge tests + slow-log suite.
Teknium