fix(startup-watchdog): bounded hard-exit escort + phase-owned progress leases
Addresses the two class-level review blockers on PR #89750: 1. Bounded hard-exit seam (escort thread). The forensic fire path (logger.critical, dump record, faulthandler, lifecycle ledger) can itself wedge — the parked main thread may hold the logging handler lock, or the disk may be full/hung. _fire() now starts an exit-escort daemon thread BEFORE any forensics; it is free of log handlers, filesystem access, module loads and application locks, and hard-exits with the restart code after _FIRE_EXIT_BOUND_S unless the normal fire path signals completion. Adversarial tests hold the logging handler lock / hang the dump write at fire time and assert the exit seam is still reached. 2. Phase-owned progress leases (report_startup_progress). Process CPU time proves process activity, not startup progress: an unrelated busy thread could extend forever while startup sits parked (false negative), and I/O-bound repair/backup accrues ~zero CPU and would be killed (false positive). Long synchronous startup phases now declare authoritative, clamped (_MAX_LEASE_S), renewable progress leases: state.db _init_schema + the version-gated data-migration chain (hermes_state_schema) and repair_state_db_schema (hermes_state) are wired. CPU progress remains only as a bounded fallback, capped at _MAX_CPU_EXTENSIONS, with leases outranking the cap. Adversarial tests cover both directions (lease saves zero-CPU legitimate work; capped CPU noise no longer hides a parked deadlock). Fire-path dump record now includes lease_count/last_lease_phase for forensics. gateway/startup_watchdog.py shim re-exports report_startup_progress. OOF-298
This commit is contained in:
@@ -18,6 +18,7 @@ from typing import Dict, Optional, Sequence
|
||||
|
||||
|
||||
from hermes_constants import get_hermes_home
|
||||
from hermes_startup_watchdog import report_startup_progress
|
||||
from hermes_state_common import (
|
||||
DEFERRED_INDEX_SQL,
|
||||
FTS_CJK_STALE_KEY,
|
||||
@@ -971,6 +972,13 @@ class SessionSchemaMixin:
|
||||
The schema_version table is retained for future data migrations
|
||||
(transforming existing rows) which cannot be handled declaratively.
|
||||
"""
|
||||
# Declare a startup-watchdog progress lease before potentially long
|
||||
# synchronous work: on multi-GB state.db files the reconciliation +
|
||||
# version-gated data migrations below are legitimately slow and can
|
||||
# be I/O-bound (near-zero CPU), which the watchdog's CPU fallback
|
||||
# would misread as a parked deadlock (OOF-298 / PR #89750).
|
||||
report_startup_progress(600.0, phase="state_db_init_schema")
|
||||
|
||||
cursor = self._conn.cursor()
|
||||
|
||||
cursor.executescript(SCHEMA_SQL)
|
||||
@@ -1068,6 +1076,9 @@ class SessionSchemaMixin:
|
||||
|
||||
else:
|
||||
current_version = row["version"] if isinstance(row, sqlite3.Row) else row[0]
|
||||
# Renew the progress lease: the version-gated chain below can
|
||||
# rewrite whole tables (PK rebuilds, backfills) on large DBs.
|
||||
report_startup_progress(600.0, phase="state_db_data_migrations")
|
||||
# Data migrations that can't be expressed declaratively (row
|
||||
# backfills, index changes tied to a specific version step) stay
|
||||
# in a version-gated chain. Column additions are handled by
|
||||
|
||||
Reference in New Issue
Block a user