SharedRouteAdapters.get called ProfileRoute.matches without guild_id, so the
documented Discord route shape (guild_id + chat_id) never authorized a cron
target and the satellite fell to standalone delivery ("DISCORD_BOT_TOKEN is
not set" every fire). A cron target has no inbound guild anchor; the route's
own guild_id is passed so its target-exact discriminators decide.
_resolve_target_transport then vetoed the authorized shared transport on the
SATELLITE's platforms.<p>.enabled (absent block or enabled: false), although
that block describes a connector the satellite never runs. The shared hit now
builds the transport directly (keeping the satellite's non-credential platform
settings), and a live native adapter with no config block is no longer read as
"disabled" (#89302) — same normalization the relay path already had.
Fixes#89302
Co-authored-by: web3blind <264741654+web3blind@users.noreply.github.com>
Five behaviour contracts, each red on origin/main: shared-bot route does not
hijack a dedicated secondary bot; secondary busy follow-up authorized against
its own allowlist; mid-turn authorization reads the admitting transport's
allowlist; shared-bot satellite resolves the primary transport for restored
sources while a downed secondary stays fail-closed; completion pre-flight runs
in the target profile's scope.
The supervised `_async_delegation_watcher` and startup-recovered process
watchers run under the ROOT scope, so a secondary profile's completion was
classified against the DEFAULT profile's state.db (row absent → "terminal" →
"permanently-gone session" warning, delivery dropped) and every durable-ledger
op (`claim`/`complete`/`release`) hit the default's ledger, stranding the real
row `pending` forever.
`_deliver_completion_notification` and `_deliver_async_delegation_group` now
run their whole pre-flight + claim + inject + settle sequence inside
`_completion_event_scope(evt)` — the runtime scope of the profile the event's
session belongs to. At multiplex startup `_restore_secondary_completion_ledgers`
replays every secondary's pending rows, which the process registry (launch
home only) never saw.
Builds on the classify-only half of #107247.
Inside a routed satellite's turn the ambient scope is the satellite's, whose
.env has no token or allowlist. Five sites still called `_is_user_authorized()`
directly there — `/topic`, the sibling-thread `/stop` grant, plugin message
injection, Discord voice transcripts and startup auto-resume — so the shared
bot's owner was refused ("not authorized to use /topic") and a satellite that
DID copy an allowlist widened who may drive those commands. They now go through
`_is_user_authorized_for_source`, and `_under_authorization_profile` derives the
transport home from the delivering adapter's owner when ingress did not stamp
one (restored/cached sources).
`_adapters_for_profile` (the resolver behind `_authorization_adapter`,
`_adapter_for_source` and now `_resolve_injection_adapter`) returns the primary
map for a shared-bot satellite: a served profile with the `{}` startup
placeholder, no reconnect pending, targeted by a default-bot route. Kanban and
cron already applied that rule; the gateway's own resolvers returned None, so
heartbeats, process completions, goal notices and delegation results for such
profiles were undeliverable after a restart. A secondary that owns a credential
(connected on any platform, or queued for reconnect) still fails closed.
`_make_default_profile_busy_session_handler` (salvaged from #105357) duplicated
the stamp-transport-home / stamp-route / resolve-home block of
`_make_default_profile_message_handler`. Both now call `_admit_primary_source`,
so the two ingress paths cannot drift apart again (#103717 was exactly that
drift).
Telegram DM chat_id == user_id for every bot, so a `chat_id` route meant to
pin a user's DM with the SHARED bot to profile `ops` also captured that
user's DM with team_b's dedicated bot: the turn, session key, state.db and
secrets were ops's while the reply left via team_b's transport.
`ProfileRoute` gains an optional `bot_profile` discriminator (None = the
default profile's bot) and `matches()` requires it to equal the receiving
adapter's owner profile. `build_source` passes `_owner_profile` and stamps it
as the fallback profile for secondary adapters; `_profile_name_for_source`
derives it from the transport ref for sources built elsewhere.
Secondary→secondary routes remain possible by naming the bot explicitly.
Salvages the discriminator idea from #105040 and #105049 in a smaller shape
(no bot-username sniffing across adapter internals; the owning profile is
already declared at `set_owner_profile`).
Co-authored-by: 0xAlyDev <agentai891@gmail.com>
Co-authored-by: Ahmett101 <Ahmett101@users.noreply.github.com>
Route primary-adapter busy callbacks through the same transport authorization scope as normal multiplexed messages, while preserving the routed profile for session state.
Adds a regression test for a Signal primary transport routed to a secondary profile.
(cherry picked from commit 429171ae0dcb78fdd20500b93c415ee13d00091f)
_make_profile_busy_session_handler stamped source.profile but never
entered _async_profile_runtime_scope before delegating, so
_is_user_authorized fell through to os.environ and read the primary
profile's allowlist. Follow-up messages from a secondary profile's
owner while their session was busy were dropped as unauthorized. Wrap
the call the same way the cold-path _make_profile_message_handler
already does.
Fixes#103717
(cherry picked from commit 760bc35c60a75fbd2500501d80c8e00d20c951e0)
_bind_api_server_session never passed profile= to set_session_vars, so
every API-server turn bound HERMES_SESSION_PROFILE="" and the terminal
tool collapsed ALL api_server sessions (default AND org profiles) onto
the shared 'default' container key. In a multiplex gateway the first
caller after restart therefore won the cache slot and org-profile turns
could reuse the default profile's sandbox container (SSH key and secrets
exposure). Pass the request profile through so org turns key their own
profile-scoped container.
(cherry picked from commit b8b01b3d8cdafab292dda4388592044393222886)
Under gateway.multiplex_profiles the default gateway serves every profile, yet
four startup/status paths still reasoned from the wrong source:
* A secondary profile's API_SERVER_KEY (which the docs REQUIRE for /p/<profile>/
auth) auto-enabled api_server in that profile's config, so
_load_secondary_profile_config raised SecondaryPortBindingConfigError and the
whole profile was skipped. gateway/config_env.py::_enable_from_env now leaves
`enabled` alone for port-binding platforms while a multiplexer loads a
NON-default profile (home override + multiplex flag, the same signal
gateway.config uses for scoped reads); the credential still lands in extra so
the shared listener can authenticate the prefix. Default profile unchanged.
* "Is this profile served?" was re-derived from the default config.yaml plus
GATEWAY_MULTIPLEX_PROFILES as seen by the CLI process. `hermes -p coder ...`
loads coder's .env, so an env-only opt-in on the default profile was invisible
(guard never fired, status said stopped) and an allowlist edit flipped the
answer before the restart. named_profile_served_by_running_multiplexer now
reads the pid-verified default gateway_state.json served_profiles (written by
_record_served_profiles) first and falls back to config derivation only when
the key is absent. The record helpers live in hermes_cli/gateway_multiplex_served.py.
* The served-profile guard ran only inside `gateway run`. `hermes -p X gateway
start|install|restart` reached the service manager, whose unit then exited 78
forever (systemd parks it while the CLI prints "started"; launchd KeepAlive
respawns every 30 s). The service verbs now run the same guard up front
(exit 78, same message) and accept --force; the Desktop /api/gateway/start
route returns 409 for a served profile instead of spawning a doomed child.
* Status surfaces disagreed: `hermes -p coder status` said stopped, `hermes -p
coder cron status` said "cron jobs will NOT fire" while `cron list` said fine,
and the default `hermes status` never listed served profiles. Both now route
through the probe / the recorded served set. The -p/--profile matcher in
_scan_gateway_pids and gateway.status._command_line_belongs_to_profile compares
the flag token for equality (`-p ops` no longer claims -- or lets `gateway
stop` SIGTERM -- an `-p ops-2` gateway).
Docs: multi-profile-gateways.md now describes the start/install refusal, the
--force flags, the API_SERVER_KEY behaviour and the single default-home
gateway_state.json (the per-profile runtime_status.json claim was wrong).
Fixes#100397
Addresses #89726#97360#71344
(cherry picked from commit d002c1864a7b6a22c53758b16b7b0cc79aea2edf)
Under gateway.multiplex_profiles every connection ledger in tools/mcp_tool.py
(_servers, _server_scope_keys/_server_tool_scopes, connecting/error/cooldown
maps, the circuit breaker, lazy schema-cache configs, trust metadata) was keyed
by the bare server NAME. The common per-tenant layout — each profile names its
server `github`/`notion` with its own token — gave only the first profile a
connection: the second profile's register_mcp_servers saw the name as "already
connected", refused to adopt it (different credentials, 4ddbcbd35e), and left
the profile silently tool-less with a healthy-looking `configured` status
(#106005 Bug 1/2, #91654). Siblings of the same bug: profile A's failing `x`
put profile B's healthy `x` into A's 10-minute connect cooldown and A's open
circuit breaker short-circuited B's calls; toolsets._resolve_toolset_memo was
not scope-keyed, so B resolved A's `mcp-<server>` tool names.
Keys are now the connection key from the new tools/mcp_tool_scope.py: the bare
name outside a multiplexer (single-profile processes are unchanged) and
(owner_scope, name) under one. Call-time lookups (_resolve_server_key) prefer
the calling scope's own connection, then a shared connection it adopted, so
identical-route profiles still share one subprocess. _select_new_servers,
the cooldown/breaker/trust maps, lazy registration and get_mcp_status all read
and write through the composite key; teardown resolves a task's key by
identity (the MCP loop has no profile context). The toolset memo key includes
registry.current_scope_key().
An owner's scoped /reload-mcp tore down its connection and, with it, every
adopting profile's tool overlay; nothing re-ran the adopters' discovery until
they reloaded. shutdown_mcp_servers(scope=) now records the orphaned adopters
and register_mcp_servers re-registers them under their own home + secret scope
once the owner's rediscovery pass completes.
Docs: multi-profile-gateways.md states the per-profile connection rule.
Fixes#106005Fixes#91654
Co-authored-by: Bergmann89 <info@bergmann89.de>
Co-authored-by: Izzy-Gottz <srulynj@gmail.com>
Keep: scopeless worker resolves the on-disk key (durability core), a keyed
build still rewrites (rotation), and the API-prefixed vault name resolves.
Drop the mock-patched gate predicate test, the inlined copy of the same
predicate, and the empty-build/empty-disk arm — they re-assert the helper's
shape rather than a behaviour contract.
Under gateway.multiplex_profiles, os.environ holds the DEFAULT profile's .env; a
secondary profile's values exist only in the per-turn secret scope. Every reader
below still read os.environ/os.getenv at call time, so a secondary profile's turn
silently used the default profile's value.
Credentials (F6): FIRECRAWL_API_KEY (read_file hosted OCR), OPENVIKING_API_KEY,
mem0-OSS OPENAI_API_KEY, MODAL_TOKEN_ID/SECRET and BROWSER_USE_API_KEY presence
gates, and the xAI video plugin's os.getenv("XAI_API_KEY") fallback AFTER the
scoped resolver had already missed — the exact fallback-after-miss shape
gateway/AGENTS.md forbids. Deleted, not re-scoped: the resolver is the scope.
Identity / tenant (F7): MEM0_USER_ID/AGENT_ID/HOST/MODE, SUPERMEMORY_CONTAINER_TAG,
RETAINDB_PROJECT, OPENVIKING_ACCOUNT/USER/AGENT (and the whole layered() env
read), HINDSIGHT_BANK_ID/MODE/retain shaping, HERMES_HONCHO_HOST. A raw read
put a secondary profile's memories into the default profile's account/bank/
project/tenant and recalled them back into the default's turns. Each now uses
get_secret with the provider's own per-profile default on a miss.
Endpoints (F8): OPENAI_BASE_URL (aux custom runtime + direct-alias expansion),
XAI_BASE_URL/HERMES_XAI_BASE_URL (aux OAuth), NOUS_INFERENCE_BASE_URL (#65941,
both the aux builder and hermes_cli.auth_nous._nous_inference_env_override),
GATEWAY_PROXY_URL (same UnscopedSecretError-only fallback shape as
GATEWAY_PROXY_KEY three lines below), FIRECRAWL_API_URL, BROWSERBASE_BASE_URL,
SUPERMEMORY/RETAINDB/HONCHO/HINDSIGHT URLs. The keys beside them were already
scoped, so a secondary's key was sent to the default profile's proxy or host.
Targets / display (F11): WEIXIN_HOME_CHANNEL (message posted into the default's
chat), HERMES_LANGUAGE, and agent/i18n's process-wide lru_cache of
display.language — now keyed by HERMES_HOME.
Outbound webhooks: hooks.outbound[].secret_env resolved from os.environ while
the gateway registers each profile's targets inside that profile's scope, so a
secondary's deliveries were signed with the default's secret or left unsigned.
Agent-cache eviction: _spawn_release_thread started a bare threading.Thread, so
commit_memory_session -> provider on_session_end ran with an EMPTY context. The
thread now runs copy_context() and, for the unscoped housekeeping sweep, enters
the owning profile's _profile_runtime_scope resolved from the session key
(agent:<profile>:...). The pressure batch does the same per key.
session_search (#82903): agent/inline_tool_executors.py::_session_search
forwarded every schema argument except `profile`, so a gateway agent could
never select a named profile's store. Forwarded; the ownership-scoping design
in #87779/#87847 is a separate design call and is not attempted here.
Live repro (/tmp/mux_audit/fix-tool-memory-reads/repro.py): 28 FAIL on
origin/main -> 0 FAIL with this change; 10 new invariant tests red on base.
Fixes#82903Fixes#65941Fixes#99121
Addresses #87779
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Michael Versluis (Berry) <michael@wve.nl>
Conflict resolution on top of main also routes MEM0_MODE / MEM0_HOST / MEM0_AGENT_ID /
MEM0_USER_ID through get_secret: identity and host are .env values like the key, and a
raw environ read hands a secondary profile the default profile's mem0 account.
(cherry picked from commit 7a5863ebafdb193d8ec2f89a7aca309774d6d69a)
The vault item is named HINDSIGHT_API_LLM_API_KEY (matching the daemon env var), while the setup wizard uses HINDSIGHT_LLM_API_KEY. Vault-fed scopes carrying only the API-prefixed name resolved empty, so the scopeless path — and the whole fail-closed machinery — never engaged on real deployments. Accept both names.
Proven end-to-end: hydrated scope now resolves the 73-char key and the rewrite gate passes.
(cherry picked from commit 2ce8c2980188ae8e4a3ed398e14fa8ed89a40644)
test_gate_refuses_keyless_build_against_keyed_disk and test_rewrite_allowed_when_build_carries_key touch no filesystem modes, so the nt skip guard was needlessly excluding them. Addresses review feedback on PR #103957.
(cherry picked from commit 196cf723571c87829be61e17df27869948db00a0)
The embedded daemon-start worker usually runs with no secret scope, so _embedded_llm_api_key resolved empty and both the profile-env compare and the upstream manager merge overwrote the file's good key with emptiness.
- _embedded_llm_api_key: explicit config, then secret scope, then on-disk profile env fallback; UnscopedSecretError treated as empty without touching os.environ (multiplex safety).
- Add _may_rewrite_profile_env gate: refuse rewrite on keyless-build plus keyed-disk, and skip the daemon stop in that case.
- Document key resolution order in plugins/memory/hindsight/README.md.
- Add 5 tests pinning disk fallback, the gate's refuse branch, rotation, keyless providers, and file survival.
(cherry picked from commit 48a4bcd536fa799b666a8d5eaae73c1816ca9e33)
goals.py now acquires its store through hermes_state_registry (one shared
writer per home), so the off-loop bootstrap tests' fake must be installed at
hermes_state_registry._open_session_db — the seam the registry documents for
tests — and accept the db_path the registry passes.
hermes_cli/goals.py cached a bare SessionDB() per HERMES_HOME. The path was
right per profile, but it was a SECOND writer connection beside the
gateway's registry handle for the same file — its own token-writer thread
and close-time checkpoint the registry knows nothing about (the #90837
corruption shape), doubled per profile under multiplexing. This is the
same topology #100682 removed from the API server. Acquire through
hermes_state_registry and drop lost-race references with release_or_close.
Three wrong-store paths in the TUI/Desktop backend under multiplexing:
- insights.get was scoped=True and then called _get_db(): before the launch
handle was pinned (#102526, #108074) a scoped first touch bound the
process-wide handle to the requested profile's state.db; even pinned, the
RPC answered for the launch profile whatever `profile` said. Route it
through _profile_db so it counts the requested profile's sessions.
- prompt.background side agents were handed the launch handle, so a
named-profile Bot Chat's bg_* rows appeared in the default profile's
session list and were missing from the profile's own history. Inherit the
parent agent's dedicated _session_db.
- session_lifecycle's gateway-owned-source guard and the notification
poller's compression-tip resolver looked the session up in the launch
store, where a named-profile row does not exist: the guard was dead and
a compression-rotated profile session lost every post-compression
delegation/background completion (the fail-closed owner gate never
matched the compressed parent's key). Both now use the session's own
store via _session_db(session).
Addresses #102157, #102526.
Two multiplex key/store mismatches in the gateway session layer:
- SessionStore._db_for_key resolved legacy agent:main keys to the AMBIENT
store. Named-profile keys were already pinned to profiles/<x>/state.db
(#97309), but a default-profile session touched inside a secondary's
runtime scope (a coalesced async-delegation drain, a cron mirror into a
default chat, a scoped watcher) was written into the secondary's state.db
— the profile_name='<A>' row inside B's store reported in #102157, after
which the routing index and row disagree and the #54878 self-heal drops a
live conversation. Default keys now resolve to _routing_home/state.db, the
launch home already pinned for the routing index. Single-profile gateways
(multiplex off) keep the ambient store byte-for-byte.
- /undo evicted the cached agent under build_session_key(source) —
agent:main:… — while the cache is keyed by the profile namespace, so on a
secondary profile the eviction missed and the next turn reused an agent
whose in-memory history still held the undone turns. Use
_session_key_for_source like /reset and /retry.
Addresses #102157.
With _parse_session_key accepting the agent:<profile>: namespace (salvaged
from #105942), the rewrite-to-agent:main workaround in
_shutdown_notification_target is dead weight: take the profile from the
parsed result like _build_process_event_source now does.
Co-authored-by: Konstantin Khlopkov <konstantin.khlopkov93@gmail.com>
`_parse_gateway_session_key()` required the literal `main` in the
namespace slot of `agent:<ns>:<platform>:<chat_type>:<chat_id>`, but
`_session_key_namespace()` (gateway/session.py) puts the multiplex
profile name there for named profiles. Every approval button click in a
named-profile QQ session therefore failed authorization ("Rejected
unauthorized approval click") and the pending approval timed out
fail-closed.
The namespace slot takes no part in any authorization decision, so the
parser now accepts any non-empty namespace; the platform and operator
checks are unchanged and pinned by new negative tests.
Fixes#98292
(cherry picked from commit 8e0f02ee770bcf931dd0adc8464f0edc185d2546)
_parse_session_key() only accepted the agent:main namespace, so under
multiplex_profiles named-profile keys (agent:<profile>:...) parsed to None and
notification/shutdown routing fell back to the LRU source cache; and
_sibling_thread_run_keys() hardcoded agent:main, so a per-user thread /stop
never matched a sibling run on a named profile. Accept any valid profile-id
namespace slot and report it as profile (main keys keep their exact historical
shape), build the sibling prefix from _session_key_namespace(source.profile),
and drop the manual agent:main re-wrap in _build_process_event_source().
(cherry picked from commit 562792015a60da3c675498edc2552a6234015457)
Users on native DeepSeek were told to pin model.context_length and
model.supports_vision in config.yaml. That is the wrong layer: the 1M
window is already in DEFAULT_CONTEXT_LENGTHS, and a global
supports_vision pin would also mark text-only deepseek-v4-pro as
multimodal.
Two catalog gaps still produced the reported symptoms:
- A leftover context_length_cache.yaml entry of 128K (the old
``deepseek`` catch-all) outlived the 1M catalog keys because
deepseek-flash was missing from _PRE_CATALOG_STALE_KEYS.
- When models.dev is empty/cold, Flash has no capability record, so
image routing falls through to lossy text. Vendor docs (2026-09-10)
mark deepseek-flash as vision-capable and deepseek-v4-pro as not.
Discard those 128K leftovers, fill Flash (and retired Flash aliases)
via _BUILTIN_MODEL_METADATA, and leave Pro catalog-only.
Reverts the in-tree org skill-marketplace: hermes_wisdom package, three
model tools, CLI/gateway/desktop/dashboard/Telegram/Slack surfaces.
Later non-Wisdom work on shared files (guest onboarding i18n, dashboard
startup schema, Slack adapter, tui_gateway) is kept; Wisdom-only call
sites and config were stripped from those files.
Stop recommending a system-RAM spill when no curated model fits resident.
Preserve explicit model selection and the existing resident quality/speed
ranking, including the separate unified-memory policy.
Require a recommendation for automatic quickstart, expose Browse when
none exists, and rename Configure to Let me choose. Keep policy copy and
reason keys consistent across the four translated local-model sections.
Cover automatic refusal and explicit spilled setup against one budget,
plus the Browse, Download and Use interactions in the desktop pane.
Same-named defaults on different connections were mislabeled as (you) in
member room-delta prompts. Compare speaker/viewer with connection source
identity so only the true self gets the suffix.
Fixes#106851
Co-authored-by: Cursor <cursoragent@cursor.com>
hermes_state emits a once-per-process SQLite-version advisory on CI's linked
3.50.4, which caplog captured as a second WARNING. Scope the assertion to the
hermes_cli.web_server logger the router actually writes to.
The dashboard polls /api/analytics/usage and /api/analytics/models every
few seconds. When state.db is malformed the read raised straight through
the handler, so uvicorn logged a full traceback at ERROR on every poll —
one fleet host wrote ~520K identical journal entries in 24 h.
Wrap both analytics handlers in `corrupt_store_as_status`: a corrupt-image
sqlite3.DatabaseError (is_malformed_db_error) becomes a 503 with an
explicit `state_db_corrupt` payload pointing at `hermes doctor`, and the
warning is gated per store path via `{path: monotonic}` (>=300 s), then
debug. Busy/locked and every other error propagate unchanged, and the file
is never renamed or quarantined from the dashboard — repair stays with
`hermes doctor` / `hermes sessions repair`. `_session_db_path_for_profile`
is split out of `_open_session_db_for_profile` so the router can name the
store without opening it.
Refs #96591
Reported-by: #96591
`hermes sessions repair --check-only` printed the corruption reason and
exited 0, so scripts and the console wrapper gating on the status read a
broken state.db as healthy. Return 1 from the CLI handler (main.py already
sys.exits a truthy return) and from the console handler, where
`_capture_output` turns the status into a ConsoleCommandError carrying the
printed reason.
Salvaged from PR #103321 (the check-only reporting part only; the probe
rewrite and connection-tracking changes were not taken). Refs #63386.
`_db_opens_cleanly` drove one probe row through the messages_fts* triggers
and rolled back. FTS5 only buffers that row in an in-memory segment until
commit, so the probe never wrote to `<fts>_idx`/`_data` and could not hit a
stale `messages_fts_trigram_idx` row waiting at the next segid — the class
where PRAGMA integrity_check, the FTS5 integrity-check command and MATCH all
report clean while every committed append fails with
`IntegrityError: constraint failed`. The probe also caught only
OperationalError; IntegrityError is a DatabaseError sibling, so even a
colliding probe would have escaped and been reported as healthy.
Now the probe issues `INSERT INTO <fts>(<fts>) VALUES('flush')` for every
FTS family inside the rolled-back transaction (capability / not-built errors
stay benign), catches sqlite3.DatabaseError, and always rolls back in a
finally. `hermes doctor` and `hermes sessions repair --check-only` surface
the corruption and `repair_state_db_schema` heals it via the FTS rebuild
strategy (verified with a real stale-segid fixture).
Refs #100227
Reported-by: #100227
The recovery commands rendered on structural corruption — the turn explainer's
`session_persistence_failed`/corrupt body, the gateway's home-channel state.db
warning, and hermes_state_repair._persistent_repair_exhausted_error — already
interpolate the active profile's state.db path, but every `hermes ...` verb in them
was bare. A bare `hermes` follows the sticky `active_profile` file, so an operator
running the pasted `hermes doctor --fix` (or `hermes sessions recover` with a
relative source) from a named-profile incident could inspect or repair a different
profile's database (#105887).
hermes_constants.profile_cli_selector() renders `-p <name> ` for a named profile
home (default home and custom roots outside the profile tree render nothing: the
default is what a bare `hermes` already means, and a custom root is only reachable
via HERMES_HOME). Every command in the three guidance sites now carries it, and
the new `fts_index` guidance inherits the same interpolation.
Live check with HERMES_HOME=<root>/profiles/research and active_profile=other:
before `1. Run \`hermes doctor --fix\`` (targets "other"); after
`1. Run \`hermes -p research doctor --fix\`` and `hermes -p research sessions
recover --source <root>/profiles/research/state.db --inspect-only`.
Refs #105887
Reported-by: Cuttingwater
`hermes doctor` reported every write-health-probe failure as "state.db FTS write
corruption" and `--fix` ran the FTS repair ladder — rebuild, REINDEX, sqlite_master
surgery + VACUUM — on the damaged file. When the damage is structural (canonical
tables/indexes), none of those rungs can fix it, each one writes to the torn file in
place, and the operator is then told to "restore from the backup copy beside
state.db": a `.malformed-backup` that is a snapshot of the same corrupt image.
`hermes sessions recover`, the tool that actually rebuilds canonical rows into a
fresh file, was never mentioned (#88587; the 1.7 GB field incident lost days to it).
Discriminate before mutating. hermes_state_repair.integrity_damage_is_structural
maps `PRAGMA integrity_check` output onto the file: a `Tree N` id resolved through
sqlite_master.rootpage, an index named in `row N missing from index X`, or a
`Freelist:` line is structural unless the object is a Hermes-owned messages_fts*
table/shadow (full-matched, so a user lookalike such as archive_fts_data is never
swept into the rebuildable set). state_db_has_structural_damage runs it read-only on
a fresh connection; an integrity_check that RAISES under the walk (torn root page)
is structural too — no FTS-only fixture does that while sessions/messages read
cleanly. doctor's state check consults it first: structural damage becomes a
manual issue naming `hermes [-p <profile>] sessions recover --source <this db>
--inspect-only` (profile pinned, #105887) and explicitly warning off the
.malformed-backup; nothing is mutated and no backup is written. FTS-only damage
keeps the existing in-place repair path.
Verified against real fixtures: a torn `sessions` root page (before: "FTS write
corruption", --fix wrote a 1:1 malformed-backup and failed; after: structural,
recover guidance, no writes) and the 16-byte DEADBEEF messages_fts_data stomp
(still repaired in place via rebuild_fts).
Salvaged from PR #88604 (liuhao1024) onto the split doctor_state.py; the
classifier lives beside the repair ladder in hermes_state_repair so the ladder
itself can consult it next.
Fixes#88587
classify_persistence_error bucketed every _DB_CORRUPTION_MARKERS hit as "corrupt",
so an error SQLite itself scoped to the FTS5 index layer (SQLITE_CORRUPT_VTAB, or an
`fts5: corrupt structure record for table "messages_fts"` report) that escaped the
write path — the detach in _enter_fts_fail_open refused (generation/lock check),
or a read/search path with no fail-open at all — reached the turn boundary and the
gateway startup notice as structural corruption: the turn ended with `.recover` /
restore-backup advice on a file whose canonical tables were provably healthy.
One provenance rule, hermes_state_errors.is_fts_scoped_corruption_error, now feeds
both the write-repair gate (SessionDB._is_fts_write_corruption_error delegates to it,
so the gateway transcript retry inherits it) and the classifier: a known result code
outranks prose (only SQLITE_CORRUPT_VTAB is FTS-scoped; bare SQLITE_CORRUPT/NOTADB
and any contradictory code fail closed), and without a code the text must both carry
a corruption marker and name a messages_fts* object. The new "fts_index" cause
renders index-scoped guidance (doctor --fix / restart, do not run recovery) in the
turn explainer and the home-channel notice. The structural fail-close is untouched:
bare malformed / not-a-database still quarantine and still classify "corrupt".
Salvaged from PR #97843 (SulthanZahran1), trimmed: the quick_check-backed
"corrupt_unconfirmed" tier is dropped — on a live handle that just observed an
unscoped SQLITE_CORRUPT, PRAGMA quick_check on a damaged shadow b-tree raises rather
than reports on 3.53.1, so the probe could never downgrade the exact shape it was
built for, and a verdict that softens quarantine guidance on prose alone weakens the
fail-close. #97841 (Finn763) reached the same fts_index cause via text markers
only; its LIKE-degradation intent already lives in _search_messages_impl (_fts_stale).
Fixes#97794
Co-authored-by: finn763 <165816600+finn763@users.noreply.github.com>