Files
hermes-agent/tests/gateway/test_handoff_watcher_multiprofile.py
T
69k4xmdfm2-blip fc5fdb8c2a fix(gateway): handoff is broken on multi-profile installs (wrong DB, wrong key, wrong bot)
`/handoff <platform>` never completes on a multiplexed gateway, and when it
does complete it can deliver through the wrong profile's bot. Three distinct
faults, all the same family: multi-profile code paths that assume a single
store / a single adapter map.

1. The watcher polls only the ROOT store.
   `_handoff_watcher` resolves `self._session_db` with no profile scope, which
   always yields the root `state.db`. But `/handoff` run under
   `hermes -p <profile>` writes `handoff_state='pending'` into THAT profile's
   store. Nothing ever reads it, so the CLI times out after 60s while the
   gateway is alive and connected. The watcher now iterates
   `[(None, None), *secondary_profiles]` and polls each inside
   `_profile_runtime_scope`.

2. The destination session key is built without the profile namespace.
   `_process_handoff` called `build_session_key()` with no `profile=`,
   producing `agent:main:...` while that profile's own adapter routes organic
   inbound messages on `agent:<profile>:...`. The handoff bound a key nobody
   reads.

3. Delivery uses the PRIMARY profile's adapter and config.
   `self.adapters` holds only the default profile's adapters (secondaries live
   in `self._profile_adapters[name]`) and `self.config` only the default's home
   channel. A secondary profile's handoff was therefore sent by the wrong bot,
   to the wrong chat, while persisting the right session key and reporting
   `handoff_state='completed'` — a false positive that looks correct in the
   database and is wrong on the wire.

Two robustness fixes in the same path:

4. Head-of-line blocking between profiles. `_process_handoff` runs a full agent
   turn plus delivery; awaiting it inline meant one slow handoff stopped the
   watcher from even polling the other profiles. With the CLI's 60s deadline, a
   valid handoff could time out purely because another profile's was ahead of
   it. Dispatch is now fire-and-forget, with an in-flight guard so a row is
   never claimed twice, and a bounded drain on shutdown.

5. Rows stranded in `running`. Only the watcher sets `running`, for the span of
   one in-process dispatch, so a row still in that state at startup belongs to
   a gateway that died mid-dispatch. It can never reach a terminal state, and
   `request_handoff` refuses new requests unless the state is
   NULL/completed/failed — that session could never hand off again, silently.
   `reclaim_stale_running_handoffs()` now fails those rows once per store at
   watcher startup. Failing (not re-queueing) is deliberate: the dead gateway
   may already have switched the session key and dispatched, so a blind retry
   risks double delivery.

Behaviour on single-profile installs is unchanged: the scope list degrades to
the unscoped root poll, `_resolve_profile_for_key` returns None when
multiplexing is off (byte-identical keys), and config/adapters fall back to
`self.config`/`self.adapters`.

Tests: 13 new across three files. Each was verified to FAIL against the
unpatched code (the fix was reverted and the suite re-run) so they are real
guards rather than decorative assertions. Verified end-to-end on a live
4-profile gateway: `handoff_state` goes failed -> completed, and a planted
stranded `running` row is reclaimed at startup with the reason recorded.
2026-08-28 11:45:19 -07:00

252 lines
8.2 KiB
Python

"""Handoff watcher must poll EVERY served profile's store, not just the root.
Regression guard for the multi-profile ``/handoff`` bug: ``/handoff`` writes
``handoff_state='pending'`` into the store of the profile the CLI ran under
(``hermes -p medicina``), while the gateway's watcher resolves ``_session_db``
from whatever HERMES_HOME is active on its task. Unscoped that is always the
ROOT store, so the pending row was never seen and the CLI timed out with the
gateway plainly alive and connected.
These tests pin the two halves of the fix:
1. the scope list includes the root (``None``) plus every multiplexed home,
and degrades to ``[None]`` for a single-profile gateway;
2. the watcher actually enters ``_profile_runtime_scope`` for each non-root
home, which is what re-points ``_session_db`` at that profile's store.
"""
import asyncio
import types
from pathlib import Path
import pytest
from gateway import run
class _FakeConfig:
def __init__(self, multiplex, allowlist=None):
self.multiplex_profiles = multiplex
self.multiplex_profile_allowlist = allowlist
def test_scopes_single_profile_gateway_is_root_only():
"""No multiplexing → exactly the legacy unscoped poll."""
runner = types.SimpleNamespace(config=_FakeConfig(multiplex=False))
assert run._handoff_watch_scopes(runner) == [(None, None)]
def test_scopes_include_root_first_then_every_secondary_home(monkeypatch):
"""Multiplexed → root first, then each SECONDARY profile as (name, home).
The default profile must NOT be yielded again: its home resolves to the
same ``state.db`` as the unscoped root poll, so repeating it would double
every tick's query count for zero benefit.
"""
homes = [
("default", Path("/h")),
("bala", Path("/h/profiles/bala")),
("medicina", Path("/h/profiles/medicina")),
]
monkeypatch.setattr(run, "_multiplex_profile_homes", lambda _cfg: homes)
runner = types.SimpleNamespace(config=_FakeConfig(multiplex=True))
scopes = run._handoff_watch_scopes(runner)
assert scopes[0] == (None, None), "root store must still be polled first"
assert scopes[1:] == [
("bala", Path("/h/profiles/bala")),
("medicina", Path("/h/profiles/medicina")),
]
assert not any(name == "default" for name, _h in scopes[1:]), (
"default profile must not be polled twice per tick"
)
def test_scopes_degrade_to_root_when_resolution_raises(monkeypatch):
"""A broken profile resolver must not disable the watcher entirely."""
def _boom(_cfg):
raise RuntimeError("profiles dir unreadable")
monkeypatch.setattr(run, "_multiplex_profile_homes", _boom)
runner = types.SimpleNamespace(config=_FakeConfig(multiplex=True))
assert run._handoff_watch_scopes(runner) == [(None, None)]
def test_scopes_tolerate_runner_without_config():
"""The watcher's own unit tests bind onto a config-less stand-in."""
assert run._handoff_watch_scopes(types.SimpleNamespace()) == [(None, None)]
class _RecordingDB:
"""Minimal AsyncSessionDB-shaped stub; records nothing pending."""
def __init__(self, tag=None):
self.polls = 0
self.tag = tag
async def list_pending_handoffs(self):
self.polls += 1
return []
@pytest.mark.asyncio
async def test_watcher_enters_profile_scope_for_each_home(monkeypatch):
"""Each non-root home is polled INSIDE ``_profile_runtime_scope``.
Entering that scope is the whole point of the fix — it is what redirects
``_session_db`` to the profile's own ``state.db``. Asserting on the scope
entries (not just the poll count) keeps the test mutation-survivable:
dropping the ``with`` still polls N times but records no scopes.
"""
scopes = [
(None, None),
("bala", Path("/h/profiles/bala")),
("medicina", Path("/h/profiles/medicina")),
]
monkeypatch.setattr(run, "_handoff_watch_scopes", lambda _r: scopes)
entered = []
class _SpyScope:
def __init__(self, home):
self.home = home
def __enter__(self):
entered.append(self.home)
return self
def __exit__(self, *exc):
return False
monkeypatch.setattr(run, "_profile_runtime_scope", _SpyScope)
async def _no_sleep(_seconds):
return None
monkeypatch.setattr(run.asyncio, "sleep", _no_sleep)
db = _RecordingDB()
states = iter([True, False])
class _Running:
def __bool__(_self):
try:
return next(states)
except StopIteration:
return False
fake = types.SimpleNamespace()
fake._session_db = db
fake._running = _Running()
async def _process_handoff(row, profile_name=None):
return None
fake._process_handoff = _process_handoff
coro = run.GatewayRunner._handoff_watcher(fake, interval=0.0)
await asyncio.wait_for(coro, timeout=5)
secondary_homes = [h for _n, h in scopes[1:]]
# Each secondary home is entered TWICE per watcher run: once by the
# startup stale-handoff reclaim, once by the poll tick. Both must be
# scoped — a reclaim outside the scope would clear the ROOT store's rows
# while reporting the profile's.
assert entered == secondary_homes * 2, (
"each secondary home must be scoped for BOTH the startup reclaim "
f"and the poll tick; got {entered}"
)
assert db.polls == 3, "root + both profiles polled once each per tick"
@pytest.mark.asyncio
async def test_each_scope_resolves_its_own_store_and_profile(monkeypatch):
"""The whole point: a DIFFERENT ``state.db`` per scope, and the profile
name reaches ``_process_handoff`` so delivery uses that profile's adapter.
The earlier test proves the ``with`` runs; it cannot prove the store was
re-resolved, because it pins one fake db for every scope. Here
``_session_db`` is a property whose value depends on the active scope, and
each store yields a pending row tagged with its profile — so a regression
that polls the root three times, or that drops ``profile_name``, fails.
"""
scopes = [
(None, None),
("bala", Path("/h/profiles/bala")),
("medicina", Path("/h/profiles/medicina")),
]
monkeypatch.setattr(run, "_handoff_watch_scopes", lambda _r: scopes)
active = {"home": None}
class _SpyScope:
def __init__(self, home):
self.home = home
def __enter__(self):
active["home"] = self.home
return self
def __exit__(self, *exc):
active["home"] = None
return False
monkeypatch.setattr(run, "_profile_runtime_scope", _SpyScope)
async def _no_sleep(_seconds):
return None
monkeypatch.setattr(run.asyncio, "sleep", _no_sleep)
class _ScopedDB:
"""Yields a row whose id identifies the store it came from."""
def __init__(self, tag):
self.tag = tag
async def list_pending_handoffs(self):
return [{"id": f"row-from-{self.tag}"}]
async def claim_handoff(self, _sid):
return True
async def complete_handoff(self, _sid):
return None
async def fail_handoff(self, _sid, _err):
return None
stores = {
None: _ScopedDB("root"),
Path("/h/profiles/bala"): _ScopedDB("bala"),
Path("/h/profiles/medicina"): _ScopedDB("medicina"),
}
processed = []
states = iter([True, False])
class _Fake:
@property
def _running(self):
try:
return next(states)
except StopIteration:
return False
@property
def _session_db(self):
# Mirrors the real property: resolves from the ACTIVE scope.
return stores[active["home"]]
async def _process_handoff(self, row, profile_name=None):
processed.append((row["id"], profile_name))
coro = run.GatewayRunner._handoff_watcher(_Fake(), interval=0.0)
await asyncio.wait_for(coro, timeout=5)
assert processed == [
("row-from-root", None),
("row-from-bala", "bala"),
("row-from-medicina", "medicina"),
], "each scope must resolve its own store AND pass its profile name through"