Files

101 lines
7.7 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# cron/ (+ kanban) — scheduled jobs and the multi-agent work queue
Applies on top of the root `AGENTS.md`. Long-form: `website/docs/developer-guide/cron-internals.md`;
user docs `website/docs/user-guide/features/cron.md`, `kanban.md`.
## Cron
`cron/jobs.py` (job store) + `cron/scheduler.py` (tick loop; `scheduler_*.py` siblings). Agents
schedule via the `cronjob` tool; users via `hermes cron list|add|edit|pause|resume|run|remove` or
`/cron`. Schedules: duration (`"30m"`, `"2h"`, `"1d"`), "every" phrase (`"every 2h"`, `"every monday
9am"`), 5-field cron (`"0 9 * * *"`), ISO one-shot (`"2026-06-01T09:00:00Z"`). Per-job fields:
`skills`, `model`/`provider` overrides, `script` (pre-run data-collection script whose stdout is
injected into the prompt; `no_agent=True` makes the script the whole job), `context_from` (chain job
A's last output into job B's prompt), `workdir` (run with that directory's `AGENTS.md`/`CLAUDE.md`
loaded), multi-platform delivery.
Hardening invariants — each guards a real failure; don't weaken without answering for it:
- **Inactivity watchdog** on cron agent sessions (`_cron_inactivity_seconds()`): default 600s idle,
`HERMES_CRON_TIMEOUT` overrides, `0` = unlimited. It is idle time, not wall-clock — a stalled
session is hard-interrupted so it cannot monopolise the scheduler, while a long-but-active job
is never cut off. Attached scripts (pre-run or `no_agent`) are bounded separately by the script
timeout (`_DEFAULT_SCRIPT_TIMEOUT`, 3600s).
- Catch-up window = half the period, clamped to 120s–2h; 120s grace for missed one-shots.
- Every recurring occurrence is accounted for: `tick()` advances `next_run_at` BEFORE dispatch
(at-most-once across a mid-run crash) and stamps `pending_slot` in the same save; a scan that
finds the stamp with a dead owner restores the instant ONCE (`cron/occurrences.py`), the
executions ledger's `scheduled_instant` blocks a second fire, `cron.catch_up_missed: false`
skips past-grace misses with a logged reason. Never drop a slot silently (#107485).
- Per-home tick lock `<home>/cron/.tick.lock` prevents duplicate ticks across processes for
that profile's store; never a `~/.hermes/...` literal.
- **The ticker binds each served profile's scope for the whole tick, including pre-loop code.**
`scheduler_provider.py::_start_multiplex` is ONE ticker iterating `profiles_to_serve()`
sequentially under `_profile_cron_scope(home)` (home + secret scope + terminal scope) — never N
threads (module globals race). Everything a tick touches lives inside that guard: store open,
lock path, backoff/failure counters (`_note_tick_failure`), job env construction, and the
`on_session_end` flush of a finished job. Supervision (`scheduler_thread.py::
SupervisedTickerThread`; start on gateway boot, stand down for homes another gateway already
serves, re-enumerate when a profile dir appears or is tombstoned) is per served home, not per
process. Why: a store opened before the scope was entered wrote a secondary profile's run
records into the launch profile's `jobs.json`.
- Cron sessions pass `skip_memory=True`; memory providers intentionally do not run during cron.
- Cron execution has its own session. Eligible continuable deliveries may mirror or seed the
reply-facing conversation: origin, origin-less home fallback, user-written bare-platform home,
or opted-in explicit targets. `all` expansions do not gain home mirror eligibility. Mirrored
briefs are labelled user turns appended at a turn boundary, preserving role alternation.
- The cron ticker runs in the desktop-spawned backend when `HERMES_DESKTOP=1` — that env var means
"spawned by the app", not "a GUI is watching" (root: capability is a property of the session).
- Background `delegate_task` is process-local; work that must survive restarts is a cron job or a
`terminal(background=True, notify_on_complete=True)` process.
## Kanban (multi-agent work queue)
Durable SQLite-backed board letting multiple profiles/workers collaborate. Users: `hermes kanban
<verb>`; dispatcher-spawned workers use a dedicated `kanban_*` toolset so their schema footprint is
zero outside a kanban task (footprint ladder rung 3).
- **CLI:** `hermes_cli/kanban.py` facade + 14 `kanban_*.py` siblings (`boards`, `db`, `db_connect`,
`db_dispatch`, `db_notify`, `db_graph` (task initialization and decomposition), `workspace`, ...). Verbs: `init, create, list (ls), show, assign, link,
unlink, comment, attach, attachments, attach-rm, complete, request-review, request-changes,
reopen-review, block, unblock, archive, tail`, plus `watch, stats, runs, log, assignees, heartbeat,
notify-*, dispatch, daemon, gc`. Argparse alias dispatch must accept both `list` and `ls` (root).
- **Toolset:** `tools/kanban_tools.py` — `kanban_show, kanban_complete, kanban_request_review,
kanban_request_changes, kanban_block, kanban_heartbeat, kanban_comment, kanban_create, kanban_link,
kanban_attach, kanban_attach_url, kanban_attachments`; platforms whose saved selection enables
`kanban` (`hermes tools enable kanban --platform <p>`; default-off, in `CONFIGURABLE_TOOLSETS`) get
the full set plus `kanban_list`/`kanban_unblock` for board routing. The check_fn reads the schema
build's own selection (`tools/kanban_toolset_context.py`), never the legacy top-level `toolsets`
key alone.
- **Dispatcher:** long-lived loop (default 60s) that reclaims stale claims, promotes ready tasks,
atomically claims, and spawns assigned profiles. Runs **inside the gateway** by default
(`kanban.dispatch_in_gateway: true`). Standalone: `plugins/kanban/systemd/hermes-kanban-dispatcher.service`.
- **Plugin assets:** `plugins/kanban/dashboard/` (web UI) + systemd unit. `kanban_db.connect` is its
own connection helper — do not alias it to `projects_db.connect` (a path-proximity generator did).
Isolation: **board** is the hard boundary — workers get `HERMES_KANBAN_BOARD` pinned in their env and
cannot see other boards; **tenant** is a soft namespace within a board (workspace-path + memory-key
isolation, one fleet serving several businesses). After `kanban.failure_limit` consecutive
non-success attempts on a task (default 2) the dispatcher auto-blocks it to stop spin loops.
Process-identity note: `kanban --preserve-cache` contains "serve" — never classify processes by argv
substring (root). Worker liveness is `(worker_pid, worker_started_at)` — the start-time fingerprint
(`gateway.status.get_process_start_time`) recorded at claim time — never bare PID existence, or a
recycled PID gets killed on reclaim.
- **Notifications leave through the task's owning profile.** `hermes_cli/kanban_db_notify.py`
subscriptions carry the profile; `gateway/kanban_watchers_notifier.py` delivers via THAT
profile's adapter under its scope (`_notify_profile_filter`), never the multiplexer's launch
adapter; a fail-closed skip logs once at WARNING with the remedy, never a bare `continue`.
Dispatched workers get `HERMES_KANBAN_BOARD` and the assignee's `HERMES_HOME` pinned in a
scrubbed child env (`build_subprocess_env` + `strip_launch_profile_env`); they never inherit the
default profile's `.env`.
- **Descendant fence is a path, not a flag.** A delegated child's Kanban marker
(`agent/delegation_context.py::DELEGATED_CHILD_ENV_MARKER`) carries the fenced board ROOT;
`kanban_path_is_fenced(path)` denies mutations only on the dispatcher-pinned `HERMES_KANBAN_DB`
or under that root, so a child working against a scratch `HERMES_HOME` keeps a writable board.
## Tests
`tests/cron/`, `tests/hermes_cli/test_kanban*.py`, `tests/tools/test_kanban*.py`. Schedule parsing
and catch-up windows are pure functions — test them as data. Never assert on the verb list or
toolset size (root: no change-detectors). Time-based tests use loose bounds (≥ 2s) and event sync.