Five behaviour contracts, each red on origin/main: shared-bot route does not
hijack a dedicated secondary bot; secondary busy follow-up authorized against
its own allowlist; mid-turn authorization reads the admitting transport's
allowlist; shared-bot satellite resolves the primary transport for restored
sources while a downed secondary stays fail-closed; completion pre-flight runs
in the target profile's scope.
The supervised `_async_delegation_watcher` and startup-recovered process
watchers run under the ROOT scope, so a secondary profile's completion was
classified against the DEFAULT profile's state.db (row absent → "terminal" →
"permanently-gone session" warning, delivery dropped) and every durable-ledger
op (`claim`/`complete`/`release`) hit the default's ledger, stranding the real
row `pending` forever.
`_deliver_completion_notification` and `_deliver_async_delegation_group` now
run their whole pre-flight + claim + inject + settle sequence inside
`_completion_event_scope(evt)` — the runtime scope of the profile the event's
session belongs to. At multiplex startup `_restore_secondary_completion_ledgers`
replays every secondary's pending rows, which the process registry (launch
home only) never saw.
Builds on the classify-only half of #107247.
Inside a routed satellite's turn the ambient scope is the satellite's, whose
.env has no token or allowlist. Five sites still called `_is_user_authorized()`
directly there — `/topic`, the sibling-thread `/stop` grant, plugin message
injection, Discord voice transcripts and startup auto-resume — so the shared
bot's owner was refused ("not authorized to use /topic") and a satellite that
DID copy an allowlist widened who may drive those commands. They now go through
`_is_user_authorized_for_source`, and `_under_authorization_profile` derives the
transport home from the delivering adapter's owner when ingress did not stamp
one (restored/cached sources).
`_adapters_for_profile` (the resolver behind `_authorization_adapter`,
`_adapter_for_source` and now `_resolve_injection_adapter`) returns the primary
map for a shared-bot satellite: a served profile with the `{}` startup
placeholder, no reconnect pending, targeted by a default-bot route. Kanban and
cron already applied that rule; the gateway's own resolvers returned None, so
heartbeats, process completions, goal notices and delegation results for such
profiles were undeliverable after a restart. A secondary that owns a credential
(connected on any platform, or queued for reconnect) still fails closed.
`_make_default_profile_busy_session_handler` (salvaged from #105357) duplicated
the stamp-transport-home / stamp-route / resolve-home block of
`_make_default_profile_message_handler`. Both now call `_admit_primary_source`,
so the two ingress paths cannot drift apart again (#103717 was exactly that
drift).
Telegram DM chat_id == user_id for every bot, so a `chat_id` route meant to
pin a user's DM with the SHARED bot to profile `ops` also captured that
user's DM with team_b's dedicated bot: the turn, session key, state.db and
secrets were ops's while the reply left via team_b's transport.
`ProfileRoute` gains an optional `bot_profile` discriminator (None = the
default profile's bot) and `matches()` requires it to equal the receiving
adapter's owner profile. `build_source` passes `_owner_profile` and stamps it
as the fallback profile for secondary adapters; `_profile_name_for_source`
derives it from the transport ref for sources built elsewhere.
Secondary→secondary routes remain possible by naming the bot explicitly.
Salvages the discriminator idea from #105040 and #105049 in a smaller shape
(no bot-username sniffing across adapter internals; the owning profile is
already declared at `set_owner_profile`).
Co-authored-by: 0xAlyDev <agentai891@gmail.com>
Co-authored-by: Ahmett101 <Ahmett101@users.noreply.github.com>
Route primary-adapter busy callbacks through the same transport authorization scope as normal multiplexed messages, while preserving the routed profile for session state.
Adds a regression test for a Signal primary transport routed to a secondary profile.
(cherry picked from commit 429171ae0dcb78fdd20500b93c415ee13d00091f)
_make_profile_busy_session_handler stamped source.profile but never
entered _async_profile_runtime_scope before delegating, so
_is_user_authorized fell through to os.environ and read the primary
profile's allowlist. Follow-up messages from a secondary profile's
owner while their session was busy were dropped as unauthorized. Wrap
the call the same way the cold-path _make_profile_message_handler
already does.
Fixes#103717
(cherry picked from commit 760bc35c60a75fbd2500501d80c8e00d20c951e0)
_bind_api_server_session never passed profile= to set_session_vars, so
every API-server turn bound HERMES_SESSION_PROFILE="" and the terminal
tool collapsed ALL api_server sessions (default AND org profiles) onto
the shared 'default' container key. In a multiplex gateway the first
caller after restart therefore won the cache slot and org-profile turns
could reuse the default profile's sandbox container (SSH key and secrets
exposure). Pass the request profile through so org turns key their own
profile-scoped container.
(cherry picked from commit b8b01b3d8cdafab292dda4388592044393222886)
Under gateway.multiplex_profiles the default gateway serves every profile, yet
four startup/status paths still reasoned from the wrong source:
* A secondary profile's API_SERVER_KEY (which the docs REQUIRE for /p/<profile>/
auth) auto-enabled api_server in that profile's config, so
_load_secondary_profile_config raised SecondaryPortBindingConfigError and the
whole profile was skipped. gateway/config_env.py::_enable_from_env now leaves
`enabled` alone for port-binding platforms while a multiplexer loads a
NON-default profile (home override + multiplex flag, the same signal
gateway.config uses for scoped reads); the credential still lands in extra so
the shared listener can authenticate the prefix. Default profile unchanged.
* "Is this profile served?" was re-derived from the default config.yaml plus
GATEWAY_MULTIPLEX_PROFILES as seen by the CLI process. `hermes -p coder ...`
loads coder's .env, so an env-only opt-in on the default profile was invisible
(guard never fired, status said stopped) and an allowlist edit flipped the
answer before the restart. named_profile_served_by_running_multiplexer now
reads the pid-verified default gateway_state.json served_profiles (written by
_record_served_profiles) first and falls back to config derivation only when
the key is absent. The record helpers live in hermes_cli/gateway_multiplex_served.py.
* The served-profile guard ran only inside `gateway run`. `hermes -p X gateway
start|install|restart` reached the service manager, whose unit then exited 78
forever (systemd parks it while the CLI prints "started"; launchd KeepAlive
respawns every 30 s). The service verbs now run the same guard up front
(exit 78, same message) and accept --force; the Desktop /api/gateway/start
route returns 409 for a served profile instead of spawning a doomed child.
* Status surfaces disagreed: `hermes -p coder status` said stopped, `hermes -p
coder cron status` said "cron jobs will NOT fire" while `cron list` said fine,
and the default `hermes status` never listed served profiles. Both now route
through the probe / the recorded served set. The -p/--profile matcher in
_scan_gateway_pids and gateway.status._command_line_belongs_to_profile compares
the flag token for equality (`-p ops` no longer claims -- or lets `gateway
stop` SIGTERM -- an `-p ops-2` gateway).
Docs: multi-profile-gateways.md now describes the start/install refusal, the
--force flags, the API_SERVER_KEY behaviour and the single default-home
gateway_state.json (the per-profile runtime_status.json claim was wrong).
Fixes#100397
Addresses #89726#97360#71344
(cherry picked from commit d002c1864a7b6a22c53758b16b7b0cc79aea2edf)
Under gateway.multiplex_profiles every connection ledger in tools/mcp_tool.py
(_servers, _server_scope_keys/_server_tool_scopes, connecting/error/cooldown
maps, the circuit breaker, lazy schema-cache configs, trust metadata) was keyed
by the bare server NAME. The common per-tenant layout — each profile names its
server `github`/`notion` with its own token — gave only the first profile a
connection: the second profile's register_mcp_servers saw the name as "already
connected", refused to adopt it (different credentials, 4ddbcbd35e), and left
the profile silently tool-less with a healthy-looking `configured` status
(#106005 Bug 1/2, #91654). Siblings of the same bug: profile A's failing `x`
put profile B's healthy `x` into A's 10-minute connect cooldown and A's open
circuit breaker short-circuited B's calls; toolsets._resolve_toolset_memo was
not scope-keyed, so B resolved A's `mcp-<server>` tool names.
Keys are now the connection key from the new tools/mcp_tool_scope.py: the bare
name outside a multiplexer (single-profile processes are unchanged) and
(owner_scope, name) under one. Call-time lookups (_resolve_server_key) prefer
the calling scope's own connection, then a shared connection it adopted, so
identical-route profiles still share one subprocess. _select_new_servers,
the cooldown/breaker/trust maps, lazy registration and get_mcp_status all read
and write through the composite key; teardown resolves a task's key by
identity (the MCP loop has no profile context). The toolset memo key includes
registry.current_scope_key().
An owner's scoped /reload-mcp tore down its connection and, with it, every
adopting profile's tool overlay; nothing re-ran the adopters' discovery until
they reloaded. shutdown_mcp_servers(scope=) now records the orphaned adopters
and register_mcp_servers re-registers them under their own home + secret scope
once the owner's rediscovery pass completes.
Docs: multi-profile-gateways.md states the per-profile connection rule.
Fixes#106005Fixes#91654
Co-authored-by: Bergmann89 <info@bergmann89.de>
Co-authored-by: Izzy-Gottz <srulynj@gmail.com>
Keep: scopeless worker resolves the on-disk key (durability core), a keyed
build still rewrites (rotation), and the API-prefixed vault name resolves.
Drop the mock-patched gate predicate test, the inlined copy of the same
predicate, and the empty-build/empty-disk arm — they re-assert the helper's
shape rather than a behaviour contract.
Under gateway.multiplex_profiles, os.environ holds the DEFAULT profile's .env; a
secondary profile's values exist only in the per-turn secret scope. Every reader
below still read os.environ/os.getenv at call time, so a secondary profile's turn
silently used the default profile's value.
Credentials (F6): FIRECRAWL_API_KEY (read_file hosted OCR), OPENVIKING_API_KEY,
mem0-OSS OPENAI_API_KEY, MODAL_TOKEN_ID/SECRET and BROWSER_USE_API_KEY presence
gates, and the xAI video plugin's os.getenv("XAI_API_KEY") fallback AFTER the
scoped resolver had already missed — the exact fallback-after-miss shape
gateway/AGENTS.md forbids. Deleted, not re-scoped: the resolver is the scope.
Identity / tenant (F7): MEM0_USER_ID/AGENT_ID/HOST/MODE, SUPERMEMORY_CONTAINER_TAG,
RETAINDB_PROJECT, OPENVIKING_ACCOUNT/USER/AGENT (and the whole layered() env
read), HINDSIGHT_BANK_ID/MODE/retain shaping, HERMES_HONCHO_HOST. A raw read
put a secondary profile's memories into the default profile's account/bank/
project/tenant and recalled them back into the default's turns. Each now uses
get_secret with the provider's own per-profile default on a miss.
Endpoints (F8): OPENAI_BASE_URL (aux custom runtime + direct-alias expansion),
XAI_BASE_URL/HERMES_XAI_BASE_URL (aux OAuth), NOUS_INFERENCE_BASE_URL (#65941,
both the aux builder and hermes_cli.auth_nous._nous_inference_env_override),
GATEWAY_PROXY_URL (same UnscopedSecretError-only fallback shape as
GATEWAY_PROXY_KEY three lines below), FIRECRAWL_API_URL, BROWSERBASE_BASE_URL,
SUPERMEMORY/RETAINDB/HONCHO/HINDSIGHT URLs. The keys beside them were already
scoped, so a secondary's key was sent to the default profile's proxy or host.
Targets / display (F11): WEIXIN_HOME_CHANNEL (message posted into the default's
chat), HERMES_LANGUAGE, and agent/i18n's process-wide lru_cache of
display.language — now keyed by HERMES_HOME.
Outbound webhooks: hooks.outbound[].secret_env resolved from os.environ while
the gateway registers each profile's targets inside that profile's scope, so a
secondary's deliveries were signed with the default's secret or left unsigned.
Agent-cache eviction: _spawn_release_thread started a bare threading.Thread, so
commit_memory_session -> provider on_session_end ran with an EMPTY context. The
thread now runs copy_context() and, for the unscoped housekeeping sweep, enters
the owning profile's _profile_runtime_scope resolved from the session key
(agent:<profile>:...). The pressure batch does the same per key.
session_search (#82903): agent/inline_tool_executors.py::_session_search
forwarded every schema argument except `profile`, so a gateway agent could
never select a named profile's store. Forwarded; the ownership-scoping design
in #87779/#87847 is a separate design call and is not attempted here.
Live repro (/tmp/mux_audit/fix-tool-memory-reads/repro.py): 28 FAIL on
origin/main -> 0 FAIL with this change; 10 new invariant tests red on base.
Fixes#82903Fixes#65941Fixes#99121
Addresses #87779
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Michael Versluis (Berry) <michael@wve.nl>
Conflict resolution on top of main also routes MEM0_MODE / MEM0_HOST / MEM0_AGENT_ID /
MEM0_USER_ID through get_secret: identity and host are .env values like the key, and a
raw environ read hands a secondary profile the default profile's mem0 account.
(cherry picked from commit 7a5863ebafdb193d8ec2f89a7aca309774d6d69a)
The vault item is named HINDSIGHT_API_LLM_API_KEY (matching the daemon env var), while the setup wizard uses HINDSIGHT_LLM_API_KEY. Vault-fed scopes carrying only the API-prefixed name resolved empty, so the scopeless path — and the whole fail-closed machinery — never engaged on real deployments. Accept both names.
Proven end-to-end: hydrated scope now resolves the 73-char key and the rewrite gate passes.
(cherry picked from commit 2ce8c2980188ae8e4a3ed398e14fa8ed89a40644)
test_gate_refuses_keyless_build_against_keyed_disk and test_rewrite_allowed_when_build_carries_key touch no filesystem modes, so the nt skip guard was needlessly excluding them. Addresses review feedback on PR #103957.
(cherry picked from commit 196cf723571c87829be61e17df27869948db00a0)
The embedded daemon-start worker usually runs with no secret scope, so _embedded_llm_api_key resolved empty and both the profile-env compare and the upstream manager merge overwrote the file's good key with emptiness.
- _embedded_llm_api_key: explicit config, then secret scope, then on-disk profile env fallback; UnscopedSecretError treated as empty without touching os.environ (multiplex safety).
- Add _may_rewrite_profile_env gate: refuse rewrite on keyless-build plus keyed-disk, and skip the daemon stop in that case.
- Document key resolution order in plugins/memory/hindsight/README.md.
- Add 5 tests pinning disk fallback, the gate's refuse branch, rotation, keyless providers, and file survival.
(cherry picked from commit 48a4bcd536fa799b666a8d5eaae73c1816ca9e33)
goals.py now acquires its store through hermes_state_registry (one shared
writer per home), so the off-loop bootstrap tests' fake must be installed at
hermes_state_registry._open_session_db — the seam the registry documents for
tests — and accept the db_path the registry passes.
hermes_cli/goals.py cached a bare SessionDB() per HERMES_HOME. The path was
right per profile, but it was a SECOND writer connection beside the
gateway's registry handle for the same file — its own token-writer thread
and close-time checkpoint the registry knows nothing about (the #90837
corruption shape), doubled per profile under multiplexing. This is the
same topology #100682 removed from the API server. Acquire through
hermes_state_registry and drop lost-race references with release_or_close.
Three wrong-store paths in the TUI/Desktop backend under multiplexing:
- insights.get was scoped=True and then called _get_db(): before the launch
handle was pinned (#102526, #108074) a scoped first touch bound the
process-wide handle to the requested profile's state.db; even pinned, the
RPC answered for the launch profile whatever `profile` said. Route it
through _profile_db so it counts the requested profile's sessions.
- prompt.background side agents were handed the launch handle, so a
named-profile Bot Chat's bg_* rows appeared in the default profile's
session list and were missing from the profile's own history. Inherit the
parent agent's dedicated _session_db.
- session_lifecycle's gateway-owned-source guard and the notification
poller's compression-tip resolver looked the session up in the launch
store, where a named-profile row does not exist: the guard was dead and
a compression-rotated profile session lost every post-compression
delegation/background completion (the fail-closed owner gate never
matched the compressed parent's key). Both now use the session's own
store via _session_db(session).
Addresses #102157, #102526.
Two multiplex key/store mismatches in the gateway session layer:
- SessionStore._db_for_key resolved legacy agent:main keys to the AMBIENT
store. Named-profile keys were already pinned to profiles/<x>/state.db
(#97309), but a default-profile session touched inside a secondary's
runtime scope (a coalesced async-delegation drain, a cron mirror into a
default chat, a scoped watcher) was written into the secondary's state.db
— the profile_name='<A>' row inside B's store reported in #102157, after
which the routing index and row disagree and the #54878 self-heal drops a
live conversation. Default keys now resolve to _routing_home/state.db, the
launch home already pinned for the routing index. Single-profile gateways
(multiplex off) keep the ambient store byte-for-byte.
- /undo evicted the cached agent under build_session_key(source) —
agent:main:… — while the cache is keyed by the profile namespace, so on a
secondary profile the eviction missed and the next turn reused an agent
whose in-memory history still held the undone turns. Use
_session_key_for_source like /reset and /retry.
Addresses #102157.
With _parse_session_key accepting the agent:<profile>: namespace (salvaged
from #105942), the rewrite-to-agent:main workaround in
_shutdown_notification_target is dead weight: take the profile from the
parsed result like _build_process_event_source now does.
Co-authored-by: Konstantin Khlopkov <konstantin.khlopkov93@gmail.com>
`_parse_gateway_session_key()` required the literal `main` in the
namespace slot of `agent:<ns>:<platform>:<chat_type>:<chat_id>`, but
`_session_key_namespace()` (gateway/session.py) puts the multiplex
profile name there for named profiles. Every approval button click in a
named-profile QQ session therefore failed authorization ("Rejected
unauthorized approval click") and the pending approval timed out
fail-closed.
The namespace slot takes no part in any authorization decision, so the
parser now accepts any non-empty namespace; the platform and operator
checks are unchanged and pinned by new negative tests.
Fixes#98292
(cherry picked from commit 8e0f02ee770bcf931dd0adc8464f0edc185d2546)
_parse_session_key() only accepted the agent:main namespace, so under
multiplex_profiles named-profile keys (agent:<profile>:...) parsed to None and
notification/shutdown routing fell back to the LRU source cache; and
_sibling_thread_run_keys() hardcoded agent:main, so a per-user thread /stop
never matched a sibling run on a named profile. Accept any valid profile-id
namespace slot and report it as profile (main keys keep their exact historical
shape), build the sibling prefix from _session_key_namespace(source.profile),
and drop the manual agent:main re-wrap in _build_process_event_source().
(cherry picked from commit 562792015a60da3c675498edc2552a6234015457)
Users on native DeepSeek were told to pin model.context_length and
model.supports_vision in config.yaml. That is the wrong layer: the 1M
window is already in DEFAULT_CONTEXT_LENGTHS, and a global
supports_vision pin would also mark text-only deepseek-v4-pro as
multimodal.
Two catalog gaps still produced the reported symptoms:
- A leftover context_length_cache.yaml entry of 128K (the old
``deepseek`` catch-all) outlived the 1M catalog keys because
deepseek-flash was missing from _PRE_CATALOG_STALE_KEYS.
- When models.dev is empty/cold, Flash has no capability record, so
image routing falls through to lossy text. Vendor docs (2026-09-10)
mark deepseek-flash as vision-capable and deepseek-v4-pro as not.
Discard those 128K leftovers, fill Flash (and retired Flash aliases)
via _BUILTIN_MODEL_METADATA, and leave Pro catalog-only.
Reverts the in-tree org skill-marketplace: hermes_wisdom package, three
model tools, CLI/gateway/desktop/dashboard/Telegram/Slack surfaces.
Later non-Wisdom work on shared files (guest onboarding i18n, dashboard
startup schema, Slack adapter, tui_gateway) is kept; Wisdom-only call
sites and config were stripped from those files.
Stop recommending a system-RAM spill when no curated model fits resident.
Preserve explicit model selection and the existing resident quality/speed
ranking, including the separate unified-memory policy.
Require a recommendation for automatic quickstart, expose Browse when
none exists, and rename Configure to Let me choose. Keep policy copy and
reason keys consistent across the four translated local-model sections.
Cover automatic refusal and explicit spilled setup against one budget,
plus the Browse, Download and Use interactions in the desktop pane.
Same-named defaults on different connections were mislabeled as (you) in
member room-delta prompts. Compare speaker/viewer with connection source
identity so only the true self gets the suffix.
Fixes#106851
Co-authored-by: Cursor <cursoragent@cursor.com>
hermes_state emits a once-per-process SQLite-version advisory on CI's linked
3.50.4, which caplog captured as a second WARNING. Scope the assertion to the
hermes_cli.web_server logger the router actually writes to.
The dashboard polls /api/analytics/usage and /api/analytics/models every
few seconds. When state.db is malformed the read raised straight through
the handler, so uvicorn logged a full traceback at ERROR on every poll —
one fleet host wrote ~520K identical journal entries in 24 h.
Wrap both analytics handlers in `corrupt_store_as_status`: a corrupt-image
sqlite3.DatabaseError (is_malformed_db_error) becomes a 503 with an
explicit `state_db_corrupt` payload pointing at `hermes doctor`, and the
warning is gated per store path via `{path: monotonic}` (>=300 s), then
debug. Busy/locked and every other error propagate unchanged, and the file
is never renamed or quarantined from the dashboard — repair stays with
`hermes doctor` / `hermes sessions repair`. `_session_db_path_for_profile`
is split out of `_open_session_db_for_profile` so the router can name the
store without opening it.
Refs #96591
Reported-by: #96591
`hermes sessions repair --check-only` printed the corruption reason and
exited 0, so scripts and the console wrapper gating on the status read a
broken state.db as healthy. Return 1 from the CLI handler (main.py already
sys.exits a truthy return) and from the console handler, where
`_capture_output` turns the status into a ConsoleCommandError carrying the
printed reason.
Salvaged from PR #103321 (the check-only reporting part only; the probe
rewrite and connection-tracking changes were not taken). Refs #63386.
`_db_opens_cleanly` drove one probe row through the messages_fts* triggers
and rolled back. FTS5 only buffers that row in an in-memory segment until
commit, so the probe never wrote to `<fts>_idx`/`_data` and could not hit a
stale `messages_fts_trigram_idx` row waiting at the next segid — the class
where PRAGMA integrity_check, the FTS5 integrity-check command and MATCH all
report clean while every committed append fails with
`IntegrityError: constraint failed`. The probe also caught only
OperationalError; IntegrityError is a DatabaseError sibling, so even a
colliding probe would have escaped and been reported as healthy.
Now the probe issues `INSERT INTO <fts>(<fts>) VALUES('flush')` for every
FTS family inside the rolled-back transaction (capability / not-built errors
stay benign), catches sqlite3.DatabaseError, and always rolls back in a
finally. `hermes doctor` and `hermes sessions repair --check-only` surface
the corruption and `repair_state_db_schema` heals it via the FTS rebuild
strategy (verified with a real stale-segid fixture).
Refs #100227
Reported-by: #100227
The recovery commands rendered on structural corruption — the turn explainer's
`session_persistence_failed`/corrupt body, the gateway's home-channel state.db
warning, and hermes_state_repair._persistent_repair_exhausted_error — already
interpolate the active profile's state.db path, but every `hermes ...` verb in them
was bare. A bare `hermes` follows the sticky `active_profile` file, so an operator
running the pasted `hermes doctor --fix` (or `hermes sessions recover` with a
relative source) from a named-profile incident could inspect or repair a different
profile's database (#105887).
hermes_constants.profile_cli_selector() renders `-p <name> ` for a named profile
home (default home and custom roots outside the profile tree render nothing: the
default is what a bare `hermes` already means, and a custom root is only reachable
via HERMES_HOME). Every command in the three guidance sites now carries it, and
the new `fts_index` guidance inherits the same interpolation.
Live check with HERMES_HOME=<root>/profiles/research and active_profile=other:
before `1. Run \`hermes doctor --fix\`` (targets "other"); after
`1. Run \`hermes -p research doctor --fix\`` and `hermes -p research sessions
recover --source <root>/profiles/research/state.db --inspect-only`.
Refs #105887
Reported-by: Cuttingwater
`hermes doctor` reported every write-health-probe failure as "state.db FTS write
corruption" and `--fix` ran the FTS repair ladder — rebuild, REINDEX, sqlite_master
surgery + VACUUM — on the damaged file. When the damage is structural (canonical
tables/indexes), none of those rungs can fix it, each one writes to the torn file in
place, and the operator is then told to "restore from the backup copy beside
state.db": a `.malformed-backup` that is a snapshot of the same corrupt image.
`hermes sessions recover`, the tool that actually rebuilds canonical rows into a
fresh file, was never mentioned (#88587; the 1.7 GB field incident lost days to it).
Discriminate before mutating. hermes_state_repair.integrity_damage_is_structural
maps `PRAGMA integrity_check` output onto the file: a `Tree N` id resolved through
sqlite_master.rootpage, an index named in `row N missing from index X`, or a
`Freelist:` line is structural unless the object is a Hermes-owned messages_fts*
table/shadow (full-matched, so a user lookalike such as archive_fts_data is never
swept into the rebuildable set). state_db_has_structural_damage runs it read-only on
a fresh connection; an integrity_check that RAISES under the walk (torn root page)
is structural too — no FTS-only fixture does that while sessions/messages read
cleanly. doctor's state check consults it first: structural damage becomes a
manual issue naming `hermes [-p <profile>] sessions recover --source <this db>
--inspect-only` (profile pinned, #105887) and explicitly warning off the
.malformed-backup; nothing is mutated and no backup is written. FTS-only damage
keeps the existing in-place repair path.
Verified against real fixtures: a torn `sessions` root page (before: "FTS write
corruption", --fix wrote a 1:1 malformed-backup and failed; after: structural,
recover guidance, no writes) and the 16-byte DEADBEEF messages_fts_data stomp
(still repaired in place via rebuild_fts).
Salvaged from PR #88604 (liuhao1024) onto the split doctor_state.py; the
classifier lives beside the repair ladder in hermes_state_repair so the ladder
itself can consult it next.
Fixes#88587
classify_persistence_error bucketed every _DB_CORRUPTION_MARKERS hit as "corrupt",
so an error SQLite itself scoped to the FTS5 index layer (SQLITE_CORRUPT_VTAB, or an
`fts5: corrupt structure record for table "messages_fts"` report) that escaped the
write path — the detach in _enter_fts_fail_open refused (generation/lock check),
or a read/search path with no fail-open at all — reached the turn boundary and the
gateway startup notice as structural corruption: the turn ended with `.recover` /
restore-backup advice on a file whose canonical tables were provably healthy.
One provenance rule, hermes_state_errors.is_fts_scoped_corruption_error, now feeds
both the write-repair gate (SessionDB._is_fts_write_corruption_error delegates to it,
so the gateway transcript retry inherits it) and the classifier: a known result code
outranks prose (only SQLITE_CORRUPT_VTAB is FTS-scoped; bare SQLITE_CORRUPT/NOTADB
and any contradictory code fail closed), and without a code the text must both carry
a corruption marker and name a messages_fts* object. The new "fts_index" cause
renders index-scoped guidance (doctor --fix / restart, do not run recovery) in the
turn explainer and the home-channel notice. The structural fail-close is untouched:
bare malformed / not-a-database still quarantine and still classify "corrupt".
Salvaged from PR #97843 (SulthanZahran1), trimmed: the quick_check-backed
"corrupt_unconfirmed" tier is dropped — on a live handle that just observed an
unscoped SQLITE_CORRUPT, PRAGMA quick_check on a damaged shadow b-tree raises rather
than reports on 3.53.1, so the probe could never downgrade the exact shape it was
built for, and a verdict that softens quarantine guidance on prose alone weakens the
fail-close. #97841 (Finn763) reached the same fts_index cause via text markers
only; its LIKE-degradation intent already lives in _search_messages_impl (_fts_stale).
Fixes#97794
Co-authored-by: finn763 <165816600+finn763@users.noreply.github.com>
optimize_fts() and vacuum() already refuse to run against a quarantined
handle (_db_corrupt / _db_replaced / _db_wal_generation_lost): both would
rewrite index/file pages in place, turning contained, diagnosable
corruption into an amplified one. rebuild_fts() never got the same guard,
despite being the more destructive of the two ("discards and recreates the
index data entirely", per its own docstring, vs. optimize_fts's segment
merge).
It's also independently reachable outside _execute_write's own quarantine
check: gateway/session_transcript.py's _rebuild_fts_once() calls
db.rebuild_fts() directly from the FTS-corruption transcript-retry path,
with no quarantine check of its own (only a WAL split-brain / foreign-holder
check, a different concern). A quarantined handle hitting that retry path
would run a full FTS rebuild — and commit it — on a corrupt, replaced, or
split-WAL-generation file.
Add the same self._raise_if_db_corrupt()/self._raise_if_db_replaced() pair
optimize_fts() already has, at the top of rebuild_fts(), before it enters
the cross-process rebuild admission.
A SIGKILL mid-write (OOM killer, #106667) can leave state.db with a garbage
page-1 header. SQLite refuses the file outright ("file is not a database",
SQLITE_NOTADB) and the sqlite3 shell's .recover opens the file like any other
client, so the lost_and_found lane failed with the same rc=26 twice although
every data page after the header survived. The #106587 quarantine now
preserves such a file as state.db.notadb-<ts>-<pid>.bak; this makes that
preserved file recoverable with `hermes sessions recover --source <bak>
--allow-partial`.
When both .recover attempts fail with "not a database", zero the 100-byte
header of the lane's private snapshot copy and rerun them. .recover trips
only on the magic check and infers page size and layout from the pages
themselves, so a zeroed header is enough; a spliced donor header (the PR's
original mechanism) instead advertises a database size / freelist that
contradicts the file and yields "database disk image is malformed" on a
direct open — verified live on a 139-page fixture, which also showed the
zeroed header recovers 60/60 sessions and 300/300 messages whether the
damage covers 100 bytes or the whole first page. The user's file is never
written; the report carries `sqlite3_cli.header_zeroed` and a warning about
the WAL boundary.
Live repro (sqlite3 shell 3.53.1 on PATH, header overwritten with random
bytes): BEFORE "page-level .recover salvage failed: ... file is not a
database (26)"; AFTER "Recovered 60 sessions and 300 messages", source md5
unchanged.
Salvaged from PR #102808 (intent; trimmed from 302 to ~40 source LOC by
dropping the donor-header/page-size sweep and the redundant preopen probe —
the shell's own refusal is the detector). Independent review on the PR by
@strzhao.
Refs #106667
Refs #106587
Reported-by: TaoMasterCoder
Cross-referenced-by: kshitijk4poor
Review feedback on #103657: the sibling clauses in the same statement
declare ESCAPE, and the trash enumeration two blocks up escapes its
underscores via replace. Use the same escaped ESCAPE form so every
underscore in the statement is a literal match instead of a
single-character wildcard. Behavior on the fixed schema is unchanged
(same enumeration split); this is consistency plus defense against
future lookalike table names.
Also reword the exclusion comment to the verified failure mechanism:
fts5 xRename renames the whole shadow family in one step, so sweeping
the cjk vtable aborts the loop on the next shadow entry and drags the
_config table (read by the vtable constructor) into the trash family.
Teknium