refactor: route middleware display events through an injected event sink (#343)
* chore: add pytest-asyncio in auto mode * test: migrate channel and stream tests to native async Convert run_async() wrapper tests to plain 'async def test_*' under pytest-asyncio auto mode. collect_events() in stream_v3_fakes becomes a coroutine awaited at every call site. * test: migrate command and model/middleware tests to native async Convert run_async() wrappers (import, alias, and fixture forms) to plain 'async def test_*'. Multi-call tests merge onto one loop as sequential awaits; none asserted on loop identity. * test: migrate TUI, notifier, gateway, and session tests to native async TUI/notifier/gateway files convert run_async wrappers to plain async tests. test_sessions.py's unittest.TestCase classes move to unittest.IsolatedAsyncioTestCase (pytest-asyncio does not await async methods on plain TestCase; converting blindly would have made ~70 tests silently vacuous). Its setUpClass keeps a one-shot asyncio.run() since IsolatedAsyncioTestCase has no async class-level hook. TestLoadingWidget in test_tui_widgets.py drops its TestCase base for the same reason. * test: replace direct asyncio.run() calls with native async tests Convert tests that called asyncio.run() (directly or via a local _run helper) to plain 'async def test_*'; delete the local helpers. * test: drop undeclared anyio markers and delete run_async helper The @pytest.mark.anyio tests relied on anyio being a transitive dep of httpx; auto-mode pytest-asyncio collects them natively. run_async() and its fixture are unreferenced after the migration, so remove them — pytest-asyncio's per-test loop teardown covers the pending-task cancellation the helper existed for (verified: full suite runs with no 'Event loop is closed' errors or destroyed-task warnings). * test: add autouse fixture for watcher cleanup * refactor: remove redundant hasattr calls * refactor: add typed middleware event sink and thread through assembly Add MiddlewareEventSink protocol + NoOpSink in middleware/events.py with a documented any-thread non-blocking contract (contract test uses a deliberately-slow fake sink). Thread an optional `events` parameter through create_cli_agent -> _get_default_middleware -> tool selector / model fallback constructors; subagent stacks are always forced to NoOpSink. * refactor: inject a notifier port into async-watcher and background middleware Add public pre_cancel_watcher() and enqueue_task_notification() to cli/async_notifier.py and a small NotifierPort protocol (middleware/notifier.py) that the module satisfies structurally. AsyncWatcherMiddleware and BackgroundExecutionMiddleware now receive the port by constructor injection at the composition root, deleting the lazy 'from ..cli import async_notifier' imports and the private _watcher_by_thread / _enqueue pokes. * refactor: invert tool-selection ownership onto a frontend event sink The adaptive tool selector now reports on_tool_selection_started / on_tool_selection / on_tool_selection_ended to the injected sink instead of writing four process-global module variables. The frontend sink (stream/sink.py FrontendEventSink) owns the selected/total/active state with consume-once + dedup-vs-last-emitted semantics; stream/tool_selection.py reads that sink object (a ToolSelectionView) rather than reaching into tool_selector's globals. Deleted: the 4 module globals, the cross-module mutations in tool_selection.py, the track_stream_selection flag, the now-vestigial _ToolSelectionTrackerMiddleware, reset_tool_selection_state_for_tests, and the autouse conftest fixture. The sink is threaded from the two interactive frontends through create_runtime_gateways -> LocalGraphGateway (read side) and _load_agent -> create_cli_agent (write side); subagent / headless stacks get NoOpSink. * refactor: route model-fallback narration through the injected event sink Delete the _ui_emit_fn / set_ui_emit module global and the ..stream.console import from model_fallback.py. The fallback middleware now reports through its injected sink: the fallback transition via the structured on_model_fallback (the frontend formats the '-> Falling back to ...' line), and the surrounding narration (primary-failure header, per-attempt outcome, exhaustion, non-fallbackable rejection) via emit_fallback_notice, preserving the exact user-facing text. The TUI binds its _append_system as the sink's fallback display where it used to call set_ui_emit (cleared on exit); the Rich CLI's sink prints to the console. _try_fallbacks / _guard_and_fallback take the sink. * refactor: declare events on the GraphGateway protocol Both gateway implementations now carry an explicit events attribute (LangGraphServerGateway holds None — no frontend renders middleware events across the HTTP boundary), so the four call sites use plain attribute access instead of getattr probing an implicit contract. * refactor: bind fallback display via the closure-scoped concrete sink The App methods used gateway.events (typed as the read-side view) and hasattr-probed for the concrete FrontendEventSink API. The enclosing factory creates that sink two hundred lines up — close over it directly: no probing, fully typed, and it becomes a constructor parameter naturally when the App class is hoisted out of the factory. * fix: end tool selection before fallback handler * fix: keep fallback display errors non-fatal * fix: preserve selector suppression for default streams * fix: restore fallback notice console display * refactor: consolidate fallback narration events * refactor: clean middleware event sink plumbing * fix: type gateway session events * refactor: make all event protocols runtime-checkable MiddlewareEventSink already carried @runtime_checkable (the stream binding guard isinstance-checks it); ToolSelectionView and SessionEvents now match, so mirroring that pattern against any of the three protocols works instead of raising TypeError. * fix(cli): close QuickJS workers after one-shot failures * fix(cli): honor no-thinking in final output * fix(channels): report failed startup accurately * fix(channels): make Telegram cleanup idempotent * fix(tui): skip command sync during exit * fix(channels): preserve startup state during retries * refactor(channels): share pending startup status * refactor(cli): expose channel startup snapshot * fix(tui): move channel startup off event loop * test(channels): release retry gate on assertion failure --------- Co-authored-by: Xi Zhang <106144707+X-iZhang@users.noreply.github.com>
This commit is contained in:
dinos
+102
-24
@@ -13,17 +13,21 @@ import pytest
|
||||
from langchain_core.exceptions import ContextOverflowError
|
||||
from langchain_core.messages import AIMessage, HumanMessage
|
||||
|
||||
from EvoScientist.middleware.events import NoOpSink
|
||||
from EvoScientist.middleware.model_fallback import (
|
||||
_guard_and_fallback,
|
||||
_is_non_fallbackable,
|
||||
_try_fallbacks,
|
||||
add_fallback,
|
||||
clear_fallbacks,
|
||||
set_ui_emit,
|
||||
)
|
||||
from EvoScientist.stream.sink import SessionEventSink
|
||||
|
||||
# ── Helpers ──────────────────────────────────────────────────────
|
||||
|
||||
# Silent sink for tests that don't assert on the fallback narration.
|
||||
_SINK = NoOpSink()
|
||||
|
||||
|
||||
def _fake_request():
|
||||
"""Build a minimal ModelRequest stub with an .override() method."""
|
||||
@@ -38,12 +42,10 @@ AI_RESPONSE = AIMessage(content="ok")
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_chain():
|
||||
"""Ensure a clean fallback chain and no UI callback for every test."""
|
||||
"""Ensure a clean fallback chain for every test."""
|
||||
clear_fallbacks()
|
||||
set_ui_emit(None)
|
||||
yield
|
||||
clear_fallbacks()
|
||||
set_ui_emit(None)
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════
|
||||
@@ -154,7 +156,7 @@ class TestTryFallbacks:
|
||||
|
||||
with patch("EvoScientist.llm.models.get_chat_model") as mock_gcm:
|
||||
mock_gcm.return_value = MagicMock()
|
||||
result = await _try_fallbacks(req, invoke, Exception("503 boom"))
|
||||
result = await _try_fallbacks(req, invoke, Exception("503 boom"), _SINK)
|
||||
|
||||
assert result is AI_RESPONSE
|
||||
invoke.assert_awaited_once()
|
||||
@@ -177,7 +179,7 @@ class TestTryFallbacks:
|
||||
|
||||
with patch("EvoScientist.llm.models.get_chat_model") as mock_gcm:
|
||||
mock_gcm.return_value = MagicMock()
|
||||
result = await _try_fallbacks(req, _invoke, Exception("503 boom"))
|
||||
result = await _try_fallbacks(req, _invoke, Exception("503 boom"), _SINK)
|
||||
|
||||
assert result is AI_RESPONSE
|
||||
assert call_count == 2
|
||||
@@ -202,7 +204,7 @@ class TestTryFallbacks:
|
||||
with patch("EvoScientist.llm.models.get_chat_model") as mock_gcm:
|
||||
mock_gcm.return_value = MagicMock()
|
||||
with pytest.raises(Exception, match="429 from fb-b") as exc_info:
|
||||
await _try_fallbacks(req, _invoke, Exception("503 primary"))
|
||||
await _try_fallbacks(req, _invoke, Exception("503 primary"), _SINK)
|
||||
|
||||
assert exc_info.value is last_error
|
||||
|
||||
@@ -218,7 +220,7 @@ class TestTryFallbacks:
|
||||
with patch("EvoScientist.llm.models.get_chat_model") as mock_gcm:
|
||||
mock_gcm.return_value = MagicMock()
|
||||
with pytest.raises(Exception, match="context_length_exceeded"):
|
||||
await _try_fallbacks(req, _invoke, Exception("503 primary"))
|
||||
await _try_fallbacks(req, _invoke, Exception("503 primary"), _SINK)
|
||||
|
||||
# get_chat_model should only have been called once (for fb-a),
|
||||
# fb-b should never be reached.
|
||||
@@ -264,7 +266,12 @@ class TestTryFallbacks:
|
||||
with patch("EvoScientist.llm.models.get_chat_model") as mock_gcm:
|
||||
mock_gcm.return_value = fallback_model
|
||||
with pytest.raises(ProviderStreamError) as exc_info:
|
||||
await _try_fallbacks(req, _invoke, Exception("openai primary failed"))
|
||||
await _try_fallbacks(
|
||||
req,
|
||||
_invoke,
|
||||
Exception("openai primary failed"),
|
||||
_SINK,
|
||||
)
|
||||
|
||||
# Attribution flipped to moonshot (the failing fallback), not
|
||||
# openai (the original request's model).
|
||||
@@ -304,7 +311,12 @@ class TestTryFallbacks:
|
||||
with patch("EvoScientist.llm.models.get_chat_model") as mock_gcm:
|
||||
mock_gcm.return_value = model
|
||||
with pytest.raises(InvalidUpdateError) as exc_info:
|
||||
await _try_fallbacks(req, _invoke, Exception("primary failed"))
|
||||
await _try_fallbacks(
|
||||
req,
|
||||
_invoke,
|
||||
Exception("primary failed"),
|
||||
_SINK,
|
||||
)
|
||||
assert exc_info.value is raised
|
||||
|
||||
|
||||
@@ -322,7 +334,9 @@ class TestGuardAndFallback:
|
||||
invoke = AsyncMock()
|
||||
|
||||
with pytest.raises(ContextOverflowError):
|
||||
await _guard_and_fallback(ContextOverflowError("overflow"), req, invoke)
|
||||
await _guard_and_fallback(
|
||||
ContextOverflowError("overflow"), req, invoke, _SINK
|
||||
)
|
||||
|
||||
invoke.assert_not_awaited()
|
||||
|
||||
@@ -349,7 +363,7 @@ class TestGuardAndFallback:
|
||||
|
||||
raised = ContextOverflowError("context length exceeded")
|
||||
with pytest.raises(ContextOverflowError) as exc_info:
|
||||
await _guard_and_fallback(raised, req, invoke)
|
||||
await _guard_and_fallback(raised, req, invoke, _SINK)
|
||||
|
||||
assert exc_info.value is raised
|
||||
invoke.assert_not_awaited()
|
||||
@@ -361,7 +375,7 @@ class TestGuardAndFallback:
|
||||
|
||||
with pytest.raises(Exception, match="invalid_request_error"):
|
||||
await _guard_and_fallback(
|
||||
Exception("400: invalid_request_error"), req, invoke
|
||||
Exception("400: invalid_request_error"), req, invoke, _SINK
|
||||
)
|
||||
|
||||
invoke.assert_not_awaited()
|
||||
@@ -373,7 +387,9 @@ class TestGuardAndFallback:
|
||||
|
||||
with patch("EvoScientist.llm.models.get_chat_model") as mock_gcm:
|
||||
mock_gcm.return_value = MagicMock()
|
||||
result = await _guard_and_fallback(Exception("503 overloaded"), req, invoke)
|
||||
result = await _guard_and_fallback(
|
||||
Exception("503 overloaded"), req, invoke, _SINK
|
||||
)
|
||||
|
||||
assert result is AI_RESPONSE
|
||||
invoke.assert_awaited_once()
|
||||
@@ -387,7 +403,7 @@ class TestGuardAndFallback:
|
||||
with patch("EvoScientist.llm.models.get_chat_model") as mock_gcm:
|
||||
mock_gcm.return_value = MagicMock()
|
||||
result = await _guard_and_fallback(
|
||||
Exception("400 Bad Request: invalid_api_key"), req, invoke
|
||||
Exception("400 Bad Request: invalid_api_key"), req, invoke, _SINK
|
||||
)
|
||||
|
||||
assert result is AI_RESPONSE
|
||||
@@ -400,35 +416,97 @@ class TestGuardAndFallback:
|
||||
|
||||
|
||||
class TestUiEmit:
|
||||
"""Verify that fallback events are surfaced via the registered callback."""
|
||||
"""Verify that fallback narration reaches the injected frontend sink.
|
||||
|
||||
The fallback middleware sends its narration lines through the same
|
||||
``fallback_display`` callback the frontend supplies, so capturing that
|
||||
callback exercises the exact user-facing text.
|
||||
"""
|
||||
|
||||
def _capturing_sink(self):
|
||||
messages: list[tuple[str, str]] = []
|
||||
sink = SessionEventSink(
|
||||
fallback_display=lambda text, style: messages.append((text, style))
|
||||
)
|
||||
return sink, messages
|
||||
|
||||
async def test_emit_captures_messages(self):
|
||||
add_fallback("fb", "prov")
|
||||
req = _fake_request()
|
||||
invoke = AsyncMock(return_value=AI_RESPONSE)
|
||||
|
||||
messages: list[tuple[str, str]] = []
|
||||
set_ui_emit(lambda text, style: messages.append((text, style)))
|
||||
sink, messages = self._capturing_sink()
|
||||
|
||||
with patch("EvoScientist.llm.models.get_chat_model") as mock_gcm:
|
||||
mock_gcm.return_value = MagicMock()
|
||||
await _try_fallbacks(req, invoke, Exception("503 down"))
|
||||
await _try_fallbacks(req, invoke, Exception("503 down"), sink)
|
||||
|
||||
texts = [t for t, _ in messages]
|
||||
assert any("Primary model failed" in t for t in texts)
|
||||
assert any("Falling back to fb (prov)" in t for t in texts)
|
||||
assert any("succeeded" in t for t in texts)
|
||||
|
||||
async def test_default_sink_prints_to_console(self):
|
||||
add_fallback("fb", "prov")
|
||||
req = _fake_request()
|
||||
invoke = AsyncMock(return_value=AI_RESPONSE)
|
||||
sink = SessionEventSink()
|
||||
|
||||
with (
|
||||
patch("EvoScientist.llm.models.get_chat_model") as mock_gcm,
|
||||
patch("EvoScientist.stream.sink.console.print") as mock_print,
|
||||
):
|
||||
mock_gcm.return_value = MagicMock()
|
||||
await _try_fallbacks(req, invoke, Exception("503 down"), sink)
|
||||
|
||||
texts = [call.args[0] for call in mock_print.call_args_list]
|
||||
assert any("Primary model failed" in t for t in texts)
|
||||
assert any("Falling back to fb (prov)" in t for t in texts)
|
||||
assert any("succeeded" in t for t in texts)
|
||||
assert all(
|
||||
call.kwargs == {"style": "yellow"} for call in mock_print.call_args_list[:2]
|
||||
)
|
||||
|
||||
async def test_display_failure_does_not_abort_fallback(self, caplog):
|
||||
add_fallback("fb", "prov")
|
||||
req = _fake_request()
|
||||
invoke = AsyncMock(return_value=AI_RESPONSE)
|
||||
sink = SessionEventSink(
|
||||
fallback_display=MagicMock(side_effect=RuntimeError("ui unavailable"))
|
||||
)
|
||||
|
||||
with (
|
||||
patch("EvoScientist.llm.models.get_chat_model") as mock_gcm,
|
||||
patch("EvoScientist.stream.sink.console.print") as mock_print,
|
||||
):
|
||||
mock_gcm.return_value = MagicMock()
|
||||
result = await _try_fallbacks(req, invoke, Exception("503 down"), sink)
|
||||
|
||||
assert result is AI_RESPONSE
|
||||
invoke.assert_awaited_once()
|
||||
assert "Fallback display callback failed" in caplog.text
|
||||
texts = [call.args[0] for call in mock_print.call_args_list]
|
||||
assert any("Primary model failed" in t for t in texts)
|
||||
assert any("Falling back to fb (prov)" in t for t in texts)
|
||||
assert any("succeeded" in t for t in texts)
|
||||
|
||||
def test_noopsink_keeps_fallback_notices_silent(self):
|
||||
sink = NoOpSink()
|
||||
|
||||
with patch("EvoScientist.stream.sink.console.print") as mock_print:
|
||||
sink.emit_fallback_notice("hidden")
|
||||
|
||||
mock_print.assert_not_called()
|
||||
|
||||
async def test_emit_shows_non_fallbackable_rejection(self):
|
||||
add_fallback("fb", "prov")
|
||||
req = _fake_request()
|
||||
invoke = AsyncMock()
|
||||
|
||||
messages: list[tuple[str, str]] = []
|
||||
set_ui_emit(lambda text, style: messages.append((text, style)))
|
||||
sink, messages = self._capturing_sink()
|
||||
|
||||
with pytest.raises(ContextOverflowError):
|
||||
await _guard_and_fallback(ContextOverflowError("overflow"), req, invoke)
|
||||
await _guard_and_fallback(
|
||||
ContextOverflowError("overflow"), req, invoke, sink
|
||||
)
|
||||
|
||||
texts = [t for t, _ in messages]
|
||||
assert any("not eligible for fallback" in t for t in texts)
|
||||
|
||||
Reference in New Issue
Block a user