bb597e1c02
* fix(gateway): pass live adapters to cron fire webhook's fire_due
The Chronos fire webhook (/api/cron/fire) called
provider.fire_due(job_id, adapters=None, loop=loop), so every
externally-triggered fire delivered through the standalone path even
with a live gateway in-process. E2EE platforms and relay-fronted
logical platforms (whose ONLY send path is the live relay adapter — no
native credential exists on the box) failed every external fire with
"platform 'X' not configured/enabled", while the same job delivered
fine under the built-in ticker (gateway/run.py passes runner.adapters).
Resolve the runner (self.gateway_runner → app['gateway_runner'] →
_gateway_runner_ref(), the same chain the drain check uses) and forward
its adapters. No runner → adapters=None, preserving the historical
standalone path byte-identically.
Note: does not by itself fix Fly-hosted scale-to-zero deployments where
NAS's callback lands on the DASHBOARD process (internal_port 9119) —
_fire_cron_job_for_profile there has no gateway runner in-process. That
topology needs a separate fire handoff (design pending).
* fix(cron): dashboard forwards Chronos fires to the gateway (503 when unreachable)
The dashboard's /api/cron/fire executed cron jobs in the DASHBOARD
process via _fire_cron_job_for_profile with adapters=None. On hosted
deployments (Fly proxy exposes only the dashboard's port) that made
every managed-cron fire deliver through the standalone send path, which
cannot serve relay-fronted logical platforms (their only sender is the
live relay adapter in the gateway process — no native credential exists
on the box) or E2EE rooms. It also ran the whole agent turn inside the
dashboard: wrong process for memory/session ownership and fire-claim
attribution.
Restore the invariant that the GATEWAY owns cron execution:
- Dashboard route: after verifying the NAS JWT and resolving the job's
profile, FORWARD the fire to the gateway api_server's own
/api/cron/fire on loopback, NAS bearer preserved (the gateway
re-verifies the JWT — defense in depth, no new trust link), and pass
the gateway's response through. Gateway unreachable → 503 so NAS
retries per the Chronos contract (non-2xx = retryable; the store CAS
de-dupes the eventual double fire). Deliberately NO local-execution
fallback.
- Endpoint resolution mirrors gateway/config.py's api_server load order
per target profile (config.yaml extra.port → API_SERVER_PORT from
process env or the profile's .env → 8642), with /p/<profile>/ prefix
routing under multiplex.
- docker/stage2-hook.sh: generate a strong API_SERVER_KEY into .env on
first boot when absent (never overwrites an operator value), so the
loopback api_server passes its startup guard on hosted images. The
fire route itself is NAS-JWT-authed; the key gates the rest of the
api_server surface. The listener binds 127.0.0.1 by default and the
Fly service exposes only the dashboard port.
- _fire_cron_job_for_profile kept but deprecated (late-binding seam
compatibility); no route calls it.
- docs/chronos-managed-cron-contract.md: document the two-hop inbound
topology and the 503-retry semantics.
Depends on the previous commit (fire webhook passes live adapters to
fire_due) — together they make NAS→dashboard→gateway fires deliver over
relay end to end.
* fix(cron): read the profile api_server port via the canonical config loader
CI guard test_config_read_guard flagged the new _gateway_fire_endpoint
for a raw yaml.safe_load of the profile's config.yaml — the exact drift
class the guard exists to kill (raw reads miss the managed-scope
overlay, ${ENV_VAR} expansion, and root-model normalization).
Read through load_config() under a HERMES_HOME override scoped to the
target profile instead (the same pattern the deprecated
_fire_cron_job_for_profile uses for its store scope), and pull the port
with cfg_get. Test updated to stub load_config rather than write a raw
config.yaml.
* fix(gateway): only messaging platforms count for the scale-to-zero arm gate
The stage2 hook now generates API_SERVER_KEY for every Docker container,
and key presence force-enables the api_server platform. The scale-to-zero
arm gate counted every enabled platform, so the loopback api_server
listener made messaging_is_relay_only_or_absent False on every hosted
instance — silently disarming the feature (the not-armed log would show
enabled platforms=['relay','api_server']).
The arm gate and the not-armed logger now share one helper that filters
to enabled MESSAGING platforms, excluding LOCAL/API_SERVER/WEBHOOK —
the same non-messaging exclusion set _connect_platforms already uses.
A genuinely enabled direct-socket platform (Discord/Telegram) still
disarms. Two of the three new tests fail without this fix.
261 lines
11 KiB
Python
261 lines
11 KiB
Python
"""Cron dashboard routes (extracted verbatim from web_server.py).
|
|
|
|
Handler bodies are byte-identical. The ``*_sync`` workers, profile resolution
|
|
and the threadpool wrapper (``_run_cron_dashboard_io``) still live in
|
|
web_server — reached via the late-binding seam in :mod:`hermes_cli.web_deps`
|
|
so ``monkeypatch.setattr(web_server, ...)`` keeps working (several cron tests
|
|
rely on exactly that).
|
|
"""
|
|
|
|
import asyncio # noqa: F401 — used by handlers
|
|
import functools # noqa: F401
|
|
import logging
|
|
from typing import Optional # noqa: F401
|
|
|
|
from fastapi import APIRouter, HTTPException, Request # noqa: F401
|
|
from fastapi.responses import JSONResponse # noqa: F401
|
|
|
|
from hermes_cli.web_deps import late
|
|
from hermes_cli.web_models import (
|
|
CronJobCreate,
|
|
CronJobUpdate,
|
|
AutomationBlueprintInstantiate,
|
|
)
|
|
|
|
# Same logger the handlers used before extraction (identical logger object).
|
|
_log = logging.getLogger("hermes_cli.web_server")
|
|
|
|
router = APIRouter()
|
|
|
|
# Late-bound web_server helpers (resolved at call time; cycle-safe,
|
|
# monkeypatch-transparent — includes config readers so existing
|
|
# ``monkeypatch.setattr(web_server, "load_config", ...)`` idioms behave
|
|
# identically for these routes).
|
|
_run_cron_dashboard_io = late("_run_cron_dashboard_io")
|
|
_list_cron_jobs_sync = late("_list_cron_jobs_sync")
|
|
_get_cron_job_sync = late("_get_cron_job_sync")
|
|
_list_cron_job_runs_sync = late("_list_cron_job_runs_sync")
|
|
_create_cron_job_sync = late("_create_cron_job_sync")
|
|
_update_cron_job_sync = late("_update_cron_job_sync")
|
|
_pause_cron_job_sync = late("_pause_cron_job_sync")
|
|
_resume_cron_job_sync = late("_resume_cron_job_sync")
|
|
_trigger_cron_job_sync = late("_trigger_cron_job_sync")
|
|
_delete_cron_job_sync = late("_delete_cron_job_sync")
|
|
_find_cron_job_profile = late("_find_cron_job_profile")
|
|
_fire_cron_job_for_profile = late("_fire_cron_job_for_profile")
|
|
_forward_cron_fire_to_gateway = late("_forward_cron_fire_to_gateway")
|
|
_call_cron_for_profile = late("_call_cron_for_profile")
|
|
_raise_if_cron_registration_error = late("_raise_if_cron_registration_error")
|
|
load_config = late("load_config")
|
|
cfg_get = late("cfg_get")
|
|
|
|
|
|
@router.get("/api/cron/jobs")
|
|
async def list_cron_jobs(profile: str = "all"):
|
|
return await _run_cron_dashboard_io(_list_cron_jobs_sync, profile)
|
|
|
|
|
|
@router.get("/api/cron/jobs/{job_id}")
|
|
async def get_cron_job(job_id: str, profile: Optional[str] = None):
|
|
return await _run_cron_dashboard_io(_get_cron_job_sync, job_id, profile)
|
|
|
|
|
|
@router.get("/api/cron/jobs/{job_id}/runs")
|
|
async def list_cron_job_runs(job_id: str, profile: Optional[str] = None, limit: int = 20):
|
|
return await _run_cron_dashboard_io(_list_cron_job_runs_sync, job_id, profile, limit)
|
|
|
|
|
|
@router.post("/api/cron/jobs")
|
|
async def create_cron_job(body: CronJobCreate, profile: Optional[str] = None):
|
|
return await _run_cron_dashboard_io(_create_cron_job_sync, body, profile)
|
|
|
|
|
|
@router.get("/api/cron/delivery-targets")
|
|
async def get_cron_delivery_targets():
|
|
"""Delivery targets the cron dropdown should offer.
|
|
|
|
Always includes the implicit ``local`` option. Beyond that, the list is
|
|
derived dynamically from the configured gateway platforms via
|
|
``cron.scheduler.cron_delivery_targets()`` — no hardcoded platform list. A
|
|
configured platform that hasn't set its cron home channel is still returned
|
|
with ``home_target_set: false`` so the UI can surface it as "configure a
|
|
home channel first" rather than hiding it.
|
|
"""
|
|
targets = [
|
|
{
|
|
"id": "local",
|
|
"name": "Local (save only)",
|
|
"home_target_set": True,
|
|
"home_env_var": None,
|
|
}
|
|
]
|
|
try:
|
|
from cron.scheduler import cron_delivery_targets
|
|
|
|
targets.extend(cron_delivery_targets())
|
|
except Exception:
|
|
_log.exception("GET /api/cron/delivery-targets failed")
|
|
return {"targets": targets}
|
|
|
|
|
|
@router.put("/api/cron/jobs/{job_id}")
|
|
async def update_cron_job(job_id: str, body: CronJobUpdate, profile: Optional[str] = None):
|
|
return await _run_cron_dashboard_io(_update_cron_job_sync, job_id, body, profile)
|
|
|
|
|
|
@router.post("/api/cron/jobs/{job_id}/pause")
|
|
async def pause_cron_job(job_id: str, profile: Optional[str] = None):
|
|
return await _run_cron_dashboard_io(_pause_cron_job_sync, job_id, profile)
|
|
|
|
|
|
@router.post("/api/cron/jobs/{job_id}/resume")
|
|
async def resume_cron_job(job_id: str, profile: Optional[str] = None):
|
|
return await _run_cron_dashboard_io(_resume_cron_job_sync, job_id, profile)
|
|
|
|
|
|
@router.post("/api/cron/jobs/{job_id}/trigger")
|
|
async def trigger_cron_job(job_id: str, profile: Optional[str] = None):
|
|
return await _run_cron_dashboard_io(_trigger_cron_job_sync, job_id, profile)
|
|
|
|
|
|
@router.delete("/api/cron/jobs/{job_id}")
|
|
async def delete_cron_job(job_id: str, profile: Optional[str] = None):
|
|
return await _run_cron_dashboard_io(_delete_cron_job_sync, job_id, profile)
|
|
|
|
|
|
@router.post("/api/cron/fire")
|
|
async def cron_fire_webhook(request: Request):
|
|
"""Chronos managed-cron fire webhook (NAS -> agent) — gateway forwarder.
|
|
|
|
Authenticated by a short-lived NAS-minted JWT (verified by the pluggable
|
|
Chronos fire-verifier), NOT the dashboard session cookie — so this path is
|
|
in ``PUBLIC_API_PATHS`` to bypass the dashboard auth gate, and the JWT is
|
|
the real gate.
|
|
|
|
The dashboard is only the PUBLIC DOOR here (on hosted deployments the Fly
|
|
proxy exposes exactly one port, the dashboard's). Cron execution belongs
|
|
to the GATEWAY process, which owns the live platform adapters — required
|
|
for relay-fronted logical platforms (their only sender is the live relay
|
|
adapter) and E2EE rooms, neither of which the dashboard's standalone send
|
|
path can serve. So after verifying the JWT this handler FORWARDS the fire
|
|
to the gateway api_server's own ``/api/cron/fire`` on loopback and passes
|
|
the gateway's response through (the gateway re-verifies the JWT — defense
|
|
in depth, no new trust link).
|
|
|
|
Gateway unreachable (scale-to-zero wake still booting, restart window,
|
|
api_server disabled) → 503, so NAS retries per the Chronos contract
|
|
(non-2xx = retryable). The store CAS claim de-dupes the eventual double
|
|
fire. Deliberately NO local-execution fallback: delivering from the wrong
|
|
process is worse than a delayed retry.
|
|
"""
|
|
from plugins.cron_providers.chronos.verify import get_fire_verifier
|
|
|
|
auth = request.headers.get("Authorization", "")
|
|
token = auth[7:].strip() if auth.startswith("Bearer ") else ""
|
|
|
|
cfg = await asyncio.to_thread(load_config)
|
|
claims = get_fire_verifier()(
|
|
token=token,
|
|
expected_audience=cfg_get(cfg, "cron", "chronos", "expected_audience", default=""),
|
|
jwks_or_key=cfg_get(cfg, "cron", "chronos", "nas_jwks_url", default="") or None,
|
|
issuer=cfg_get(cfg, "cron", "chronos", "portal_url", default="") or None,
|
|
)
|
|
if claims is None:
|
|
return JSONResponse({"error": "invalid fire token"}, status_code=401)
|
|
|
|
try:
|
|
body = await request.json()
|
|
except Exception:
|
|
body = {}
|
|
job_id = (body or {}).get("job_id") if isinstance(body, dict) else None
|
|
if not job_id:
|
|
return JSONResponse({"error": "missing job_id"}, status_code=400)
|
|
|
|
# _find_cron_job_profile walks every profile and lists its jobs (file
|
|
# I/O per profile) — run it off the event loop like the other cron
|
|
# dashboard endpoints.
|
|
profile = await _run_cron_dashboard_io(_find_cron_job_profile, job_id)
|
|
if not profile:
|
|
# Job is gone (cancelled / completed) — nothing to fire. 200 so NAS
|
|
# does not retry a fire that is intentionally absent.
|
|
return JSONResponse({"status": "gone", "job_id": job_id}, status_code=200)
|
|
|
|
forwarded = await _forward_cron_fire_to_gateway(profile, job_id, auth)
|
|
if forwarded is None:
|
|
return JSONResponse(
|
|
{
|
|
"error": "gateway unreachable; retry",
|
|
"job_id": job_id,
|
|
"profile": profile,
|
|
},
|
|
status_code=503,
|
|
)
|
|
status_code, gateway_body = forwarded
|
|
if isinstance(gateway_body, dict):
|
|
gateway_body.setdefault("job_id", job_id)
|
|
return JSONResponse(gateway_body, status_code=status_code)
|
|
|
|
|
|
@router.get("/api/cron/blueprints")
|
|
async def list_cron_blueprints():
|
|
"""Return the blueprint catalog as form schemas for the dashboard gallery.
|
|
|
|
The ``deliver`` slot's options are rewritten from the user's actually
|
|
configured gateway platforms (plus the universal origin/local/all), so the
|
|
form never offers a platform that isn't connected.
|
|
"""
|
|
try:
|
|
from cron.blueprint_catalog import CATALOG, blueprint_catalog_entry
|
|
|
|
deliver_options = None
|
|
try:
|
|
from cron.scheduler import cron_delivery_targets
|
|
|
|
platforms = [t["id"] for t in cron_delivery_targets() if t.get("id")]
|
|
deliver_options = ["origin", "local", *platforms]
|
|
except Exception:
|
|
_log.debug("cron_delivery_targets unavailable; using static deliver options", exc_info=True)
|
|
|
|
entries = []
|
|
for r in CATALOG:
|
|
entry = blueprint_catalog_entry(r)
|
|
if deliver_options:
|
|
for f in entry.get("fields", []):
|
|
if f.get("name") == "deliver":
|
|
f["options"] = deliver_options
|
|
entries.append(entry)
|
|
return {"blueprints": entries}
|
|
except Exception as e:
|
|
_log.exception("GET /api/cron/blueprints failed")
|
|
raise HTTPException(status_code=500, detail=str(e))
|
|
|
|
|
|
@router.post("/api/cron/blueprints/instantiate")
|
|
async def instantiate_blueprint(body: AutomationBlueprintInstantiate, profile: str = "default"):
|
|
"""Fill a blueprint's slots and create the cron job (form-submit path)."""
|
|
try:
|
|
from cron.blueprint_catalog import fill_blueprint, get_blueprint, BlueprintFillError
|
|
|
|
blueprint = get_blueprint(body.blueprint)
|
|
if blueprint is None:
|
|
raise HTTPException(status_code=404, detail=f"Unknown blueprint: {body.blueprint}")
|
|
try:
|
|
spec = fill_blueprint(blueprint, body.values)
|
|
except BlueprintFillError as exc:
|
|
# Field-level validation error — 422 so the form can show it inline.
|
|
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
|
# Blueprint-created jobs deliver to the dashboard's configured target by
|
|
# default; the form's deliver slot overrides via spec["deliver"].
|
|
spec.pop("origin", None)
|
|
# create_job does per-profile file I/O — keep it off the event loop
|
|
# like the sibling cron endpoints (partial avoids **spec keys ever
|
|
# colliding with the wrapper's own parameters).
|
|
_create = functools.partial(_call_cron_for_profile, profile, "create_job", **spec)
|
|
return await _run_cron_dashboard_io(_create)
|
|
except HTTPException:
|
|
raise
|
|
except Exception as e:
|
|
_raise_if_cron_registration_error(e)
|
|
_log.exception("POST /api/cron/blueprints/instantiate failed")
|
|
raise HTTPException(status_code=400, detail=str(e))
|