Files
hermes-agent/hermes_cli/web_routers/cron.py
T
Ben Barclay bb597e1c02 fix(cron): managed-cron fires execute in the gateway process (live adapters + dashboard forwarder) (#84339)
* fix(gateway): pass live adapters to cron fire webhook's fire_due

The Chronos fire webhook (/api/cron/fire) called
provider.fire_due(job_id, adapters=None, loop=loop), so every
externally-triggered fire delivered through the standalone path even
with a live gateway in-process. E2EE platforms and relay-fronted
logical platforms (whose ONLY send path is the live relay adapter — no
native credential exists on the box) failed every external fire with
"platform 'X' not configured/enabled", while the same job delivered
fine under the built-in ticker (gateway/run.py passes runner.adapters).

Resolve the runner (self.gateway_runner → app['gateway_runner'] →
_gateway_runner_ref(), the same chain the drain check uses) and forward
its adapters. No runner → adapters=None, preserving the historical
standalone path byte-identically.

Note: does not by itself fix Fly-hosted scale-to-zero deployments where
NAS's callback lands on the DASHBOARD process (internal_port 9119) —
_fire_cron_job_for_profile there has no gateway runner in-process. That
topology needs a separate fire handoff (design pending).

* fix(cron): dashboard forwards Chronos fires to the gateway (503 when unreachable)

The dashboard's /api/cron/fire executed cron jobs in the DASHBOARD
process via _fire_cron_job_for_profile with adapters=None. On hosted
deployments (Fly proxy exposes only the dashboard's port) that made
every managed-cron fire deliver through the standalone send path, which
cannot serve relay-fronted logical platforms (their only sender is the
live relay adapter in the gateway process — no native credential exists
on the box) or E2EE rooms. It also ran the whole agent turn inside the
dashboard: wrong process for memory/session ownership and fire-claim
attribution.

Restore the invariant that the GATEWAY owns cron execution:

- Dashboard route: after verifying the NAS JWT and resolving the job's
  profile, FORWARD the fire to the gateway api_server's own
  /api/cron/fire on loopback, NAS bearer preserved (the gateway
  re-verifies the JWT — defense in depth, no new trust link), and pass
  the gateway's response through. Gateway unreachable → 503 so NAS
  retries per the Chronos contract (non-2xx = retryable; the store CAS
  de-dupes the eventual double fire). Deliberately NO local-execution
  fallback.
- Endpoint resolution mirrors gateway/config.py's api_server load order
  per target profile (config.yaml extra.port → API_SERVER_PORT from
  process env or the profile's .env → 8642), with /p/<profile>/ prefix
  routing under multiplex.
- docker/stage2-hook.sh: generate a strong API_SERVER_KEY into .env on
  first boot when absent (never overwrites an operator value), so the
  loopback api_server passes its startup guard on hosted images. The
  fire route itself is NAS-JWT-authed; the key gates the rest of the
  api_server surface. The listener binds 127.0.0.1 by default and the
  Fly service exposes only the dashboard port.
- _fire_cron_job_for_profile kept but deprecated (late-binding seam
  compatibility); no route calls it.
- docs/chronos-managed-cron-contract.md: document the two-hop inbound
  topology and the 503-retry semantics.

Depends on the previous commit (fire webhook passes live adapters to
fire_due) — together they make NAS→dashboard→gateway fires deliver over
relay end to end.

* fix(cron): read the profile api_server port via the canonical config loader

CI guard test_config_read_guard flagged the new _gateway_fire_endpoint
for a raw yaml.safe_load of the profile's config.yaml — the exact drift
class the guard exists to kill (raw reads miss the managed-scope
overlay, ${ENV_VAR} expansion, and root-model normalization).

Read through load_config() under a HERMES_HOME override scoped to the
target profile instead (the same pattern the deprecated
_fire_cron_job_for_profile uses for its store scope), and pull the port
with cfg_get. Test updated to stub load_config rather than write a raw
config.yaml.

* fix(gateway): only messaging platforms count for the scale-to-zero arm gate

The stage2 hook now generates API_SERVER_KEY for every Docker container,
and key presence force-enables the api_server platform. The scale-to-zero
arm gate counted every enabled platform, so the loopback api_server
listener made messaging_is_relay_only_or_absent False on every hosted
instance — silently disarming the feature (the not-armed log would show
enabled platforms=['relay','api_server']).

The arm gate and the not-armed logger now share one helper that filters
to enabled MESSAGING platforms, excluding LOCAL/API_SERVER/WEBHOOK —
the same non-messaging exclusion set _connect_platforms already uses.
A genuinely enabled direct-socket platform (Discord/Telegram) still
disarms. Two of the three new tests fail without this fix.
2026-08-12 17:04:44 +10:00

261 lines
11 KiB
Python

"""Cron dashboard routes (extracted verbatim from web_server.py).
Handler bodies are byte-identical. The ``*_sync`` workers, profile resolution
and the threadpool wrapper (``_run_cron_dashboard_io``) still live in
web_server — reached via the late-binding seam in :mod:`hermes_cli.web_deps`
so ``monkeypatch.setattr(web_server, ...)`` keeps working (several cron tests
rely on exactly that).
"""
import asyncio # noqa: F401 — used by handlers
import functools # noqa: F401
import logging
from typing import Optional # noqa: F401
from fastapi import APIRouter, HTTPException, Request # noqa: F401
from fastapi.responses import JSONResponse # noqa: F401
from hermes_cli.web_deps import late
from hermes_cli.web_models import (
CronJobCreate,
CronJobUpdate,
AutomationBlueprintInstantiate,
)
# Same logger the handlers used before extraction (identical logger object).
_log = logging.getLogger("hermes_cli.web_server")
router = APIRouter()
# Late-bound web_server helpers (resolved at call time; cycle-safe,
# monkeypatch-transparent — includes config readers so existing
# ``monkeypatch.setattr(web_server, "load_config", ...)`` idioms behave
# identically for these routes).
_run_cron_dashboard_io = late("_run_cron_dashboard_io")
_list_cron_jobs_sync = late("_list_cron_jobs_sync")
_get_cron_job_sync = late("_get_cron_job_sync")
_list_cron_job_runs_sync = late("_list_cron_job_runs_sync")
_create_cron_job_sync = late("_create_cron_job_sync")
_update_cron_job_sync = late("_update_cron_job_sync")
_pause_cron_job_sync = late("_pause_cron_job_sync")
_resume_cron_job_sync = late("_resume_cron_job_sync")
_trigger_cron_job_sync = late("_trigger_cron_job_sync")
_delete_cron_job_sync = late("_delete_cron_job_sync")
_find_cron_job_profile = late("_find_cron_job_profile")
_fire_cron_job_for_profile = late("_fire_cron_job_for_profile")
_forward_cron_fire_to_gateway = late("_forward_cron_fire_to_gateway")
_call_cron_for_profile = late("_call_cron_for_profile")
_raise_if_cron_registration_error = late("_raise_if_cron_registration_error")
load_config = late("load_config")
cfg_get = late("cfg_get")
@router.get("/api/cron/jobs")
async def list_cron_jobs(profile: str = "all"):
return await _run_cron_dashboard_io(_list_cron_jobs_sync, profile)
@router.get("/api/cron/jobs/{job_id}")
async def get_cron_job(job_id: str, profile: Optional[str] = None):
return await _run_cron_dashboard_io(_get_cron_job_sync, job_id, profile)
@router.get("/api/cron/jobs/{job_id}/runs")
async def list_cron_job_runs(job_id: str, profile: Optional[str] = None, limit: int = 20):
return await _run_cron_dashboard_io(_list_cron_job_runs_sync, job_id, profile, limit)
@router.post("/api/cron/jobs")
async def create_cron_job(body: CronJobCreate, profile: Optional[str] = None):
return await _run_cron_dashboard_io(_create_cron_job_sync, body, profile)
@router.get("/api/cron/delivery-targets")
async def get_cron_delivery_targets():
"""Delivery targets the cron dropdown should offer.
Always includes the implicit ``local`` option. Beyond that, the list is
derived dynamically from the configured gateway platforms via
``cron.scheduler.cron_delivery_targets()`` — no hardcoded platform list. A
configured platform that hasn't set its cron home channel is still returned
with ``home_target_set: false`` so the UI can surface it as "configure a
home channel first" rather than hiding it.
"""
targets = [
{
"id": "local",
"name": "Local (save only)",
"home_target_set": True,
"home_env_var": None,
}
]
try:
from cron.scheduler import cron_delivery_targets
targets.extend(cron_delivery_targets())
except Exception:
_log.exception("GET /api/cron/delivery-targets failed")
return {"targets": targets}
@router.put("/api/cron/jobs/{job_id}")
async def update_cron_job(job_id: str, body: CronJobUpdate, profile: Optional[str] = None):
return await _run_cron_dashboard_io(_update_cron_job_sync, job_id, body, profile)
@router.post("/api/cron/jobs/{job_id}/pause")
async def pause_cron_job(job_id: str, profile: Optional[str] = None):
return await _run_cron_dashboard_io(_pause_cron_job_sync, job_id, profile)
@router.post("/api/cron/jobs/{job_id}/resume")
async def resume_cron_job(job_id: str, profile: Optional[str] = None):
return await _run_cron_dashboard_io(_resume_cron_job_sync, job_id, profile)
@router.post("/api/cron/jobs/{job_id}/trigger")
async def trigger_cron_job(job_id: str, profile: Optional[str] = None):
return await _run_cron_dashboard_io(_trigger_cron_job_sync, job_id, profile)
@router.delete("/api/cron/jobs/{job_id}")
async def delete_cron_job(job_id: str, profile: Optional[str] = None):
return await _run_cron_dashboard_io(_delete_cron_job_sync, job_id, profile)
@router.post("/api/cron/fire")
async def cron_fire_webhook(request: Request):
"""Chronos managed-cron fire webhook (NAS -> agent) — gateway forwarder.
Authenticated by a short-lived NAS-minted JWT (verified by the pluggable
Chronos fire-verifier), NOT the dashboard session cookie — so this path is
in ``PUBLIC_API_PATHS`` to bypass the dashboard auth gate, and the JWT is
the real gate.
The dashboard is only the PUBLIC DOOR here (on hosted deployments the Fly
proxy exposes exactly one port, the dashboard's). Cron execution belongs
to the GATEWAY process, which owns the live platform adapters — required
for relay-fronted logical platforms (their only sender is the live relay
adapter) and E2EE rooms, neither of which the dashboard's standalone send
path can serve. So after verifying the JWT this handler FORWARDS the fire
to the gateway api_server's own ``/api/cron/fire`` on loopback and passes
the gateway's response through (the gateway re-verifies the JWT — defense
in depth, no new trust link).
Gateway unreachable (scale-to-zero wake still booting, restart window,
api_server disabled) → 503, so NAS retries per the Chronos contract
(non-2xx = retryable). The store CAS claim de-dupes the eventual double
fire. Deliberately NO local-execution fallback: delivering from the wrong
process is worse than a delayed retry.
"""
from plugins.cron_providers.chronos.verify import get_fire_verifier
auth = request.headers.get("Authorization", "")
token = auth[7:].strip() if auth.startswith("Bearer ") else ""
cfg = await asyncio.to_thread(load_config)
claims = get_fire_verifier()(
token=token,
expected_audience=cfg_get(cfg, "cron", "chronos", "expected_audience", default=""),
jwks_or_key=cfg_get(cfg, "cron", "chronos", "nas_jwks_url", default="") or None,
issuer=cfg_get(cfg, "cron", "chronos", "portal_url", default="") or None,
)
if claims is None:
return JSONResponse({"error": "invalid fire token"}, status_code=401)
try:
body = await request.json()
except Exception:
body = {}
job_id = (body or {}).get("job_id") if isinstance(body, dict) else None
if not job_id:
return JSONResponse({"error": "missing job_id"}, status_code=400)
# _find_cron_job_profile walks every profile and lists its jobs (file
# I/O per profile) — run it off the event loop like the other cron
# dashboard endpoints.
profile = await _run_cron_dashboard_io(_find_cron_job_profile, job_id)
if not profile:
# Job is gone (cancelled / completed) — nothing to fire. 200 so NAS
# does not retry a fire that is intentionally absent.
return JSONResponse({"status": "gone", "job_id": job_id}, status_code=200)
forwarded = await _forward_cron_fire_to_gateway(profile, job_id, auth)
if forwarded is None:
return JSONResponse(
{
"error": "gateway unreachable; retry",
"job_id": job_id,
"profile": profile,
},
status_code=503,
)
status_code, gateway_body = forwarded
if isinstance(gateway_body, dict):
gateway_body.setdefault("job_id", job_id)
return JSONResponse(gateway_body, status_code=status_code)
@router.get("/api/cron/blueprints")
async def list_cron_blueprints():
"""Return the blueprint catalog as form schemas for the dashboard gallery.
The ``deliver`` slot's options are rewritten from the user's actually
configured gateway platforms (plus the universal origin/local/all), so the
form never offers a platform that isn't connected.
"""
try:
from cron.blueprint_catalog import CATALOG, blueprint_catalog_entry
deliver_options = None
try:
from cron.scheduler import cron_delivery_targets
platforms = [t["id"] for t in cron_delivery_targets() if t.get("id")]
deliver_options = ["origin", "local", *platforms]
except Exception:
_log.debug("cron_delivery_targets unavailable; using static deliver options", exc_info=True)
entries = []
for r in CATALOG:
entry = blueprint_catalog_entry(r)
if deliver_options:
for f in entry.get("fields", []):
if f.get("name") == "deliver":
f["options"] = deliver_options
entries.append(entry)
return {"blueprints": entries}
except Exception as e:
_log.exception("GET /api/cron/blueprints failed")
raise HTTPException(status_code=500, detail=str(e))
@router.post("/api/cron/blueprints/instantiate")
async def instantiate_blueprint(body: AutomationBlueprintInstantiate, profile: str = "default"):
"""Fill a blueprint's slots and create the cron job (form-submit path)."""
try:
from cron.blueprint_catalog import fill_blueprint, get_blueprint, BlueprintFillError
blueprint = get_blueprint(body.blueprint)
if blueprint is None:
raise HTTPException(status_code=404, detail=f"Unknown blueprint: {body.blueprint}")
try:
spec = fill_blueprint(blueprint, body.values)
except BlueprintFillError as exc:
# Field-level validation error — 422 so the form can show it inline.
raise HTTPException(status_code=422, detail=str(exc)) from exc
# Blueprint-created jobs deliver to the dashboard's configured target by
# default; the form's deliver slot overrides via spec["deliver"].
spec.pop("origin", None)
# create_job does per-profile file I/O — keep it off the event loop
# like the sibling cron endpoints (partial avoids **spec keys ever
# colliding with the wrapper's own parameters).
_create = functools.partial(_call_cron_for_profile, profile, "create_job", **spec)
return await _run_cron_dashboard_io(_create)
except HTTPException:
raise
except Exception as e:
_raise_if_cron_registration_error(e)
_log.exception("POST /api/cron/blueprints/instantiate failed")
raise HTTPException(status_code=400, detail=str(e))