fix(startup-watchdog): bounded hard-exit escort + phase-owned progress leases

Addresses the two class-level review blockers on PR #89750:

1. Bounded hard-exit seam (escort thread). The forensic fire path
   (logger.critical, dump record, faulthandler, lifecycle ledger) can
   itself wedge — the parked main thread may hold the logging handler
   lock, or the disk may be full/hung. _fire() now starts an exit-escort
   daemon thread BEFORE any forensics; it is free of log handlers,
   filesystem access, module loads and application locks, and hard-exits
   with the restart code after _FIRE_EXIT_BOUND_S unless the normal fire
   path signals completion. Adversarial tests hold the logging handler
   lock / hang the dump write at fire time and assert the exit seam is
   still reached.

2. Phase-owned progress leases (report_startup_progress). Process CPU
   time proves process activity, not startup progress: an unrelated busy
   thread could extend forever while startup sits parked (false
   negative), and I/O-bound repair/backup accrues ~zero CPU and would be
   killed (false positive). Long synchronous startup phases now declare
   authoritative, clamped (_MAX_LEASE_S), renewable progress leases:
   state.db _init_schema + the version-gated data-migration chain
   (hermes_state_schema) and repair_state_db_schema (hermes_state) are
   wired. CPU progress remains only as a bounded fallback, capped at
   _MAX_CPU_EXTENSIONS, with leases outranking the cap. Adversarial
   tests cover both directions (lease saves zero-CPU legitimate work;
   capped CPU noise no longer hides a parked deadlock).

Fire-path dump record now includes lease_count/last_lease_phase for
forensics. gateway/startup_watchdog.py shim re-exports
report_startup_progress.

OOF-298
This commit is contained in:
Shannon Sands
2026-08-20 12:09:12 +10:00
committed by Teknium
parent f5bb1e144d
commit 852db61abe
5 changed files with 423 additions and 25 deletions
+11
View File
@@ -18,6 +18,7 @@ from typing import Dict, Optional, Sequence
from hermes_constants import get_hermes_home
from hermes_startup_watchdog import report_startup_progress
from hermes_state_common import (
DEFERRED_INDEX_SQL,
FTS_CJK_STALE_KEY,
@@ -971,6 +972,13 @@ class SessionSchemaMixin:
The schema_version table is retained for future data migrations
(transforming existing rows) which cannot be handled declaratively.
"""
# Declare a startup-watchdog progress lease before potentially long
# synchronous work: on multi-GB state.db files the reconciliation +
# version-gated data migrations below are legitimately slow and can
# be I/O-bound (near-zero CPU), which the watchdog's CPU fallback
# would misread as a parked deadlock (OOF-298 / PR #89750).
report_startup_progress(600.0, phase="state_db_init_schema")
cursor = self._conn.cursor()
cursor.executescript(SCHEMA_SQL)
@@ -1068,6 +1076,9 @@ class SessionSchemaMixin:
else:
current_version = row["version"] if isinstance(row, sqlite3.Row) else row[0]
# Renew the progress lease: the version-gated chain below can
# rewrite whole tables (PK rebuilds, backfills) on large DBs.
report_startup_progress(600.0, phase="state_db_data_migrations")
# Data migrations that can't be expressed declaratively (row
# backfills, index changes tied to a specific version step) stay
# in a version-gated chain. Column additions are handled by