kshitijk4poor
db339f0051
A gateway process opened state.db from ~12 call sites, each minting its own writer connection, self._lock, close-time WAL checkpoint, and token-writer thread. With N independent writers on one WAL file, one connection's close-time checkpoint could race another's growth — the lost/reordered-page-write signature across 11+ incidents (#90837). Adds hermes_state_registry.py: a process-wide, per-path, refcounted shared registry owning the writer boundary. - acquire(path): same resolved path returns the same instance (one writer connection, one lock, one token-writer thread) for every long-lived in-process caller (gateway runner, SessionStore, per-agent lazy recall, cron per-job, mirror, channel_directory, slash_commands, shutdown_flush, session_search, react_to_message, delegate, mcp_serve, auto_archive, tui_gateway). - close() on a shared instance is a NO-OP — the registry owns the lifecycle, so one caller's close can never tear down a writer other callers still hold. - Generation-aware retirement on inode change: a replaced state.db RETIRES the live generation (never lent again) but keeps it alive for existing holders; release is object-keyed so holders of the old generation drain it independently of the new one. The old generation's own write path still fails with the typed StateDbReplacedError (existing protection, unchanged). - Replacement-open failure leaves NO registry entry for the path — the next acquire retries fresh, never hands out a closed stale object. - All teardown runs OUTSIDE the registry lock: a final release's WAL checkpoint can never stall acquisition for every state.db. - close_shared_session_dbs() at gateway shutdown drains every generation (live + retired) as the final safety net. CLI one-shots, recovery flows, and read-only cross-profile opens keep using SessionDB() directly with their own close() — only long-lived in-process sites route through the registry. References #90837 (root-cause tracker stays open: the #10 EOF signature and the WAL-lifecycle A/B verdict remain under investigation there).
63 lines
2.4 KiB
Python
63 lines
2.4 KiB
Python
"""Tests for KeyboardInterrupt handling in exit cleanup paths.
|
|
|
|
``except Exception`` does not catch ``KeyboardInterrupt`` (which inherits
|
|
from ``BaseException``). A second Ctrl+C during exit cleanup must not
|
|
abort remaining cleanup steps. These tests exercise the actual production
|
|
code paths — not a copy of the try/except pattern.
|
|
"""
|
|
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
import pytest
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _mock_runtime_provider(monkeypatch):
|
|
"""run_job calls resolve_runtime_provider which can try real network
|
|
auto-detection (~4s of socket timeouts in hermetic CI). Mock it out
|
|
since these tests don't care about provider resolution — the agent
|
|
is mocked too."""
|
|
import hermes_cli.runtime_provider as rp
|
|
def _fake_resolve(*args, **kwargs):
|
|
return {
|
|
"provider": "openrouter",
|
|
"api_key": "test-key",
|
|
"base_url": "https://openrouter.ai/api/v1",
|
|
"model": "test/model",
|
|
"api_mode": "chat_completions",
|
|
}
|
|
monkeypatch.setattr(rp, "resolve_runtime_provider", _fake_resolve)
|
|
|
|
|
|
class TestCronJobCleanup:
|
|
"""cron/scheduler.py — end_session + close in the finally block."""
|
|
|
|
def test_keyboard_interrupt_in_end_session_does_not_skip_close(self):
|
|
"""If end_session raises KeyboardInterrupt, close() must still run."""
|
|
mock_db = MagicMock()
|
|
mock_db.end_session.side_effect = KeyboardInterrupt
|
|
|
|
from cron import scheduler
|
|
|
|
job = {
|
|
"id": "test-job-1",
|
|
"name": "test cleanup",
|
|
"prompt": "hello",
|
|
"schedule": "0 9 * * *",
|
|
"model": "test/model",
|
|
}
|
|
|
|
with patch("hermes_state.get_shared_session_db", return_value=mock_db), \
|
|
patch.object(scheduler, "_build_job_prompt", return_value="hello"), \
|
|
patch.object(scheduler, "_resolve_origin", return_value=None), \
|
|
patch.object(scheduler, "_resolve_delivery_target", return_value=None), \
|
|
patch("dotenv.load_dotenv", return_value=None), \
|
|
patch("run_agent.AIAgent") as MockAgent:
|
|
# Make the agent raise immediately so we hit the finally block
|
|
MockAgent.return_value.run_conversation.side_effect = RuntimeError("boom")
|
|
scheduler.run_job(job)
|
|
|
|
mock_db.end_session.assert_called_once()
|
|
mock_db.close.assert_called_once()
|
|
|