Ben Barclay
bb597e1c02
* fix(gateway): pass live adapters to cron fire webhook's fire_due
The Chronos fire webhook (/api/cron/fire) called
provider.fire_due(job_id, adapters=None, loop=loop), so every
externally-triggered fire delivered through the standalone path even
with a live gateway in-process. E2EE platforms and relay-fronted
logical platforms (whose ONLY send path is the live relay adapter — no
native credential exists on the box) failed every external fire with
"platform 'X' not configured/enabled", while the same job delivered
fine under the built-in ticker (gateway/run.py passes runner.adapters).
Resolve the runner (self.gateway_runner → app['gateway_runner'] →
_gateway_runner_ref(), the same chain the drain check uses) and forward
its adapters. No runner → adapters=None, preserving the historical
standalone path byte-identically.
Note: does not by itself fix Fly-hosted scale-to-zero deployments where
NAS's callback lands on the DASHBOARD process (internal_port 9119) —
_fire_cron_job_for_profile there has no gateway runner in-process. That
topology needs a separate fire handoff (design pending).
* fix(cron): dashboard forwards Chronos fires to the gateway (503 when unreachable)
The dashboard's /api/cron/fire executed cron jobs in the DASHBOARD
process via _fire_cron_job_for_profile with adapters=None. On hosted
deployments (Fly proxy exposes only the dashboard's port) that made
every managed-cron fire deliver through the standalone send path, which
cannot serve relay-fronted logical platforms (their only sender is the
live relay adapter in the gateway process — no native credential exists
on the box) or E2EE rooms. It also ran the whole agent turn inside the
dashboard: wrong process for memory/session ownership and fire-claim
attribution.
Restore the invariant that the GATEWAY owns cron execution:
- Dashboard route: after verifying the NAS JWT and resolving the job's
profile, FORWARD the fire to the gateway api_server's own
/api/cron/fire on loopback, NAS bearer preserved (the gateway
re-verifies the JWT — defense in depth, no new trust link), and pass
the gateway's response through. Gateway unreachable → 503 so NAS
retries per the Chronos contract (non-2xx = retryable; the store CAS
de-dupes the eventual double fire). Deliberately NO local-execution
fallback.
- Endpoint resolution mirrors gateway/config.py's api_server load order
per target profile (config.yaml extra.port → API_SERVER_PORT from
process env or the profile's .env → 8642), with /p/<profile>/ prefix
routing under multiplex.
- docker/stage2-hook.sh: generate a strong API_SERVER_KEY into .env on
first boot when absent (never overwrites an operator value), so the
loopback api_server passes its startup guard on hosted images. The
fire route itself is NAS-JWT-authed; the key gates the rest of the
api_server surface. The listener binds 127.0.0.1 by default and the
Fly service exposes only the dashboard port.
- _fire_cron_job_for_profile kept but deprecated (late-binding seam
compatibility); no route calls it.
- docs/chronos-managed-cron-contract.md: document the two-hop inbound
topology and the 503-retry semantics.
Depends on the previous commit (fire webhook passes live adapters to
fire_due) — together they make NAS→dashboard→gateway fires deliver over
relay end to end.
* fix(cron): read the profile api_server port via the canonical config loader
CI guard test_config_read_guard flagged the new _gateway_fire_endpoint
for a raw yaml.safe_load of the profile's config.yaml — the exact drift
class the guard exists to kill (raw reads miss the managed-scope
overlay, ${ENV_VAR} expansion, and root-model normalization).
Read through load_config() under a HERMES_HOME override scoped to the
target profile instead (the same pattern the deprecated
_fire_cron_job_for_profile uses for its store scope), and pull the port
with cfg_get. Test updated to stub load_config rather than write a raw
config.yaml.
* fix(gateway): only messaging platforms count for the scale-to-zero arm gate
The stage2 hook now generates API_SERVER_KEY for every Docker container,
and key presence force-enables the api_server platform. The scale-to-zero
arm gate counted every enabled platform, so the loopback api_server
listener made messaging_is_relay_only_or_absent False on every hosted
instance — silently disarming the feature (the not-armed log would show
enabled platforms=['relay','api_server']).
The arm gate and the not-armed logger now share one helper that filters
to enabled MESSAGING platforms, excluding LOCAL/API_SERVER/WEBHOOK —
the same non-messaging exclusion set _connect_platforms already uses.
A genuinely enabled direct-socket platform (Discord/Telegram) still
disarms. Two of the three new tests fail without this fix.
331 lines
12 KiB
Python
331 lines
12 KiB
Python
"""Tests for the Chronos cron-fire webhook (POST /api/cron/fire) — Phase 4E.2.
|
|
|
|
The webhook authenticates a NAS-minted JWT via the pluggable fire-verifier
|
|
(NOT API_SERVER_KEY), then runs the job via the resolved provider's fire_due in
|
|
the background, returning 202. These tests monkeypatch the verifier and
|
|
resolve_cron_scheduler — the verifier itself is tested with real crypto in
|
|
test_chronos_verify.py.
|
|
"""
|
|
|
|
import asyncio
|
|
import threading
|
|
from types import SimpleNamespace
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
from aiohttp import web
|
|
from aiohttp.test_utils import TestClient, TestServer
|
|
|
|
from gateway.config import PlatformConfig
|
|
from gateway.platforms.api_server import APIServerAdapter, cors_middleware
|
|
|
|
_MOD = "gateway.platforms.api_server"
|
|
|
|
|
|
def _make_adapter() -> APIServerAdapter:
|
|
return APIServerAdapter(PlatformConfig(enabled=True, extra={"key": "sk-secret"}))
|
|
|
|
|
|
def _create_app(adapter: APIServerAdapter) -> web.Application:
|
|
app = web.Application(middlewares=[cors_middleware])
|
|
app["api_server_adapter"] = adapter
|
|
app.router.add_post("/api/cron/fire", adapter._handle_cron_fire)
|
|
return app
|
|
|
|
|
|
@pytest.fixture
|
|
def adapter():
|
|
return _make_adapter()
|
|
|
|
|
|
class _SpyProvider:
|
|
"""Records fire_due calls; stands in for the resolved provider."""
|
|
|
|
def __init__(self):
|
|
self.fired = []
|
|
|
|
def fire_due(self, job_id, *, adapters=None, loop=None):
|
|
self.fired.append(job_id)
|
|
return True
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_valid_fire_reservation_blocks_drain_before_body_and_task(adapter, monkeypatch):
|
|
runner = SimpleNamespace(_draining=False, _external_drain_active=False)
|
|
body_started = asyncio.Event()
|
|
release_body = asyncio.Event()
|
|
fired = threading.Event()
|
|
release_fire = threading.Event()
|
|
|
|
class BlockingProvider:
|
|
def fire_due(self, job_id, *, adapters=None, loop=None):
|
|
fired.set()
|
|
release_fire.wait(timeout=2)
|
|
return True
|
|
|
|
original_json = web.Request.json
|
|
|
|
async def delayed_json(request):
|
|
body_started.set()
|
|
await release_body.wait()
|
|
return await original_json(request)
|
|
|
|
monkeypatch.setattr("cron.scheduler_provider.resolve_cron_scheduler", BlockingProvider)
|
|
monkeypatch.setattr(
|
|
"plugins.cron_providers.chronos.verify.get_fire_verifier",
|
|
lambda: (lambda **kw: {"purpose": "cron_fire"}),
|
|
)
|
|
app = _create_app(adapter)
|
|
with patch("gateway.run._gateway_runner_ref", lambda: runner), patch.object(
|
|
web.Request, "json", delayed_json
|
|
):
|
|
async with TestClient(TestServer(app)) as cli:
|
|
request_task = asyncio.create_task(
|
|
cli.post(
|
|
"/api/cron/fire",
|
|
headers={"Authorization": "Bearer good"},
|
|
json={"job_id": "abc123"},
|
|
)
|
|
)
|
|
await body_started.wait()
|
|
assert adapter.active_agent_work_count() == 1
|
|
|
|
release_body.set()
|
|
response = await request_task
|
|
assert response.status == 202
|
|
await asyncio.to_thread(fired.wait, 2)
|
|
assert adapter.active_agent_work_count() == 1
|
|
release_fire.set()
|
|
for _ in range(50):
|
|
if adapter.active_agent_work_count() == 0:
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
|
|
assert adapter.active_agent_work_count() == 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_missing_job_id_400(adapter, monkeypatch):
|
|
"""Valid token but no job_id → 400, no fire."""
|
|
spy = _SpyProvider()
|
|
monkeypatch.setattr("cron.scheduler_provider.resolve_cron_scheduler", lambda: spy)
|
|
monkeypatch.setattr(
|
|
"plugins.cron_providers.chronos.verify.get_fire_verifier",
|
|
lambda: (lambda **kw: {"purpose": "cron_fire"}),
|
|
)
|
|
|
|
app = _create_app(adapter)
|
|
async with TestClient(TestServer(app)) as cli:
|
|
resp = await cli.post("/api/cron/fire",
|
|
headers={"Authorization": "Bearer good"},
|
|
json={})
|
|
assert resp.status == 400
|
|
assert spy.fired == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_fire_does_not_require_api_server_key(adapter, monkeypatch):
|
|
"""The fire endpoint must NOT gate on API_SERVER_KEY — auth is the NAS-JWT.
|
|
A request with NO API key header but a valid fire token still succeeds."""
|
|
spy = _SpyProvider()
|
|
monkeypatch.setattr("cron.scheduler_provider.resolve_cron_scheduler", lambda: spy)
|
|
monkeypatch.setattr(
|
|
"plugins.cron_providers.chronos.verify.get_fire_verifier",
|
|
lambda: (lambda **kw: {"purpose": "cron_fire"}),
|
|
)
|
|
|
|
app = _create_app(adapter)
|
|
async with TestClient(TestServer(app)) as cli:
|
|
# Bearer is the FIRE token, not the API_SERVER_KEY "sk-secret".
|
|
resp = await cli.post("/api/cron/fire",
|
|
headers={"Authorization": "Bearer nas-jwt"},
|
|
json={"job_id": "j9"})
|
|
assert resp.status == 202
|
|
for _ in range(50):
|
|
if spy.fired:
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
assert spy.fired == ["j9"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_sync_verifier_runs_off_the_event_loop(adapter, monkeypatch):
|
|
"""The verifier resolves the signing key from a JWKS URL — a synchronous
|
|
HTTP GET on a cache miss. It must run via asyncio.to_thread, NOT inline on
|
|
the event loop, or a slow/rate-limited portal stalls every other adapter
|
|
sharing the loop. Proof: the sync verifier executes on a worker thread, not
|
|
the loop thread.
|
|
"""
|
|
loop_thread_id = threading.get_ident()
|
|
seen = {}
|
|
|
|
def blocking_verifier(**kw):
|
|
seen["thread_id"] = threading.get_ident()
|
|
return {"purpose": "cron_fire"}
|
|
|
|
spy = _SpyProvider()
|
|
monkeypatch.setattr("cron.scheduler_provider.resolve_cron_scheduler", lambda: spy)
|
|
monkeypatch.setattr(
|
|
"plugins.cron_providers.chronos.verify.get_fire_verifier",
|
|
lambda: blocking_verifier,
|
|
)
|
|
|
|
app = _create_app(adapter)
|
|
async with TestClient(TestServer(app)) as cli:
|
|
resp = await cli.post("/api/cron/fire",
|
|
headers={"Authorization": "Bearer good"},
|
|
json={"job_id": "off-loop"})
|
|
assert resp.status == 202
|
|
|
|
# If the verifier had run inline on the loop, its thread id would equal the
|
|
# loop thread's; to_thread puts it on a distinct worker thread.
|
|
assert seen["thread_id"] != loop_thread_id
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_crashing_verifier_fails_closed_401(adapter, monkeypatch):
|
|
"""A verifier that raises must be treated as a rejection (401), never admit
|
|
the fire, and never surface as a 500 — this is the only inbound that can
|
|
trigger remote job execution, so it fails closed.
|
|
"""
|
|
spy = _SpyProvider()
|
|
monkeypatch.setattr("cron.scheduler_provider.resolve_cron_scheduler", lambda: spy)
|
|
|
|
def exploding_verifier(**kw):
|
|
raise RuntimeError("JWKS endpoint unreachable")
|
|
|
|
monkeypatch.setattr(
|
|
"plugins.cron_providers.chronos.verify.get_fire_verifier",
|
|
lambda: exploding_verifier,
|
|
)
|
|
|
|
app = _create_app(adapter)
|
|
async with TestClient(TestServer(app)) as cli:
|
|
resp = await cli.post("/api/cron/fire",
|
|
headers={"Authorization": "Bearer boom"},
|
|
json={"job_id": "abc123"})
|
|
assert resp.status == 401
|
|
|
|
await asyncio.sleep(0.05)
|
|
assert spy.fired == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_async_verifier_is_awaited(adapter, monkeypatch):
|
|
"""A coroutine verifier (a future async escape-hatch) is awaited directly
|
|
rather than dispatched to a thread — a valid async verify still fires.
|
|
"""
|
|
spy = _SpyProvider()
|
|
monkeypatch.setattr("cron.scheduler_provider.resolve_cron_scheduler", lambda: spy)
|
|
|
|
async def async_verifier(**kw):
|
|
return {"purpose": "cron_fire", "aud": "agent:x"}
|
|
|
|
monkeypatch.setattr(
|
|
"plugins.cron_providers.chronos.verify.get_fire_verifier",
|
|
lambda: async_verifier,
|
|
)
|
|
|
|
app = _create_app(adapter)
|
|
async with TestClient(TestServer(app)) as cli:
|
|
resp = await cli.post("/api/cron/fire",
|
|
headers={"Authorization": "Bearer good"},
|
|
json={"job_id": "async-ok"})
|
|
assert resp.status == 202
|
|
|
|
for _ in range(50):
|
|
if spy.fired:
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
assert spy.fired == ["async-ok"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_fire_passes_live_adapters_to_provider(adapter, monkeypatch):
|
|
"""The fire webhook must hand the gateway's live adapters to fire_due —
|
|
delivery parity with the built-in ticker (gateway/run.py passes
|
|
runner.adapters). Without them, relay-fronted logical platforms (whose
|
|
ONLY send path is the live relay adapter — no native credential exists on
|
|
the box) and E2EE platforms fail every external-provider fire with
|
|
"platform 'X' not configured/enabled" while the same job delivers fine
|
|
under the in-process ticker."""
|
|
seen = {}
|
|
|
|
class _AdapterSpyProvider:
|
|
def fire_due(self, job_id, *, adapters=None, loop=None):
|
|
seen["job_id"] = job_id
|
|
seen["adapters"] = adapters
|
|
seen["loop"] = loop
|
|
return True
|
|
|
|
live_adapters = {"relay": object()}
|
|
runner = SimpleNamespace(
|
|
_draining=False,
|
|
_external_drain_active=False,
|
|
adapters=live_adapters,
|
|
)
|
|
|
|
monkeypatch.setattr(
|
|
"cron.scheduler_provider.resolve_cron_scheduler",
|
|
lambda: _AdapterSpyProvider(),
|
|
)
|
|
monkeypatch.setattr(
|
|
"plugins.cron_providers.chronos.verify.get_fire_verifier",
|
|
lambda: (lambda **kw: {"purpose": "cron_fire"}),
|
|
)
|
|
|
|
with patch("gateway.run._gateway_runner_ref", lambda: runner):
|
|
app = _create_app(adapter)
|
|
async with TestClient(TestServer(app)) as cli:
|
|
resp = await cli.post("/api/cron/fire",
|
|
headers={"Authorization": "Bearer good"},
|
|
json={"job_id": "with-adapters"})
|
|
assert resp.status == 202
|
|
|
|
for _ in range(50):
|
|
if seen:
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
|
|
assert seen.get("job_id") == "with-adapters"
|
|
assert seen.get("adapters") is live_adapters
|
|
assert seen.get("loop") is not None
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_fire_without_runner_passes_none_adapters(adapter, monkeypatch):
|
|
"""No gateway runner (standalone/edge case) → fire still works with
|
|
adapters=None, preserving the historical standalone delivery path."""
|
|
seen = {}
|
|
|
|
class _AdapterSpyProvider:
|
|
def fire_due(self, job_id, *, adapters=None, loop=None):
|
|
seen["job_id"] = job_id
|
|
seen["adapters"] = adapters
|
|
return True
|
|
|
|
monkeypatch.setattr(
|
|
"cron.scheduler_provider.resolve_cron_scheduler",
|
|
lambda: _AdapterSpyProvider(),
|
|
)
|
|
monkeypatch.setattr(
|
|
"plugins.cron_providers.chronos.verify.get_fire_verifier",
|
|
lambda: (lambda **kw: {"purpose": "cron_fire"}),
|
|
)
|
|
|
|
with patch("gateway.run._gateway_runner_ref", lambda: None):
|
|
app = _create_app(adapter)
|
|
async with TestClient(TestServer(app)) as cli:
|
|
resp = await cli.post("/api/cron/fire",
|
|
headers={"Authorization": "Bearer good"},
|
|
json={"job_id": "no-runner"})
|
|
assert resp.status == 202
|
|
|
|
for _ in range(50):
|
|
if seen:
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
|
|
assert seen.get("job_id") == "no-runner"
|
|
assert seen.get("adapters") is None
|