Coatue FR (Frank Long): jobs delivering into shared channels publish
engine failure notices ('⚠️ Cron X failed…') to those channels with no
opt-out. Adds an optional per-job failure_deliver field sharing
deliver's grammar: on failure, targets resolve from failure_deliver
when set (local = structural silence; state still recorded in
last_status/last_error/run history). Success delivery is unchanged;
absent field = today's behavior byte-for-byte.
Honored by every failure-category engine notice: the run_job failure
summary (+streak nudge), the escaped-failure retry path, drift-skip and
blocked-config alerts (composed into the same delivery), and the
gateway-shutdown interrupted-run notice (_notify_interrupted_cron_jobs).
Surfaces: cronjob tool create/update (same bot-chat validation as
deliver; '' clears on update), hermes cron create/edit
--failure-deliver, docs tip in automate-with-cron.
Existing fake_deliver test doubles gained **kwargs for the new
for_failure keyword — signature-compat only, no behavior change.
* feat(cron): durable failure incidents with signature dedup and ack
Introduce a durable cron incident store (cron_incidents in the shared
cron/executions.db) that groups "same job + same error signature" across
runs, so a known recurring failure stops re-pinging the operator every run
once it has been acknowledged.
- cron/incidents.py: lazily-created incident table (detected -> alerted ->
reviewed -> closed lifecycle; closed is per-signature terminal), sha256
signature dedup over job_id + normalized error, redacted/truncated error
storage, failure-type classification, and ack/list/get/count helpers.
- cron/scheduler.py: record an incident on the failure delivery path and
suppress the per-run failure ping when the exact signature is acked (both
the normal failure path and the processing-raised retry path). Best-effort:
an incident-store error never breaks the cron run or delivery. Streak nudge,
alert-once markers, and delivery-error behavior are untouched.
- hermes_cli: add `hermes cron incidents [--state ...]` and
`hermes cron incidents ack <id>`.
- tests/cron/test_cron_incidents.py: dedup, lifecycle, redaction,
classification, lazy-schema, scheduler gating, and CLI coverage.
Non-goals deferred to later slices: Discord buttons/review view, HMAC action
tokens, owner-agent review launch, approval-gated fixes, incident playbooks.
* refactor(cron): tighten incident lifecycle, wire alerted state and suppressed_acked outcome
Follow-ups on top of the salvaged #94692:
- Drop the dead 'reviewed' state and the SQLite CHECK (state validity
lives in INCIDENT_STATES so future slices can add states without a
table rebuild); lifecycle is detected -> alerted -> closed.
- Actually mark incidents 'alerted' after a failure ping reaches
delivery, on both the normal and exception delivery paths.
- Record ack-suppressed runs with a distinct 'suppressed_acked'
delivery outcome (registered in cron_health monitoring) instead of
the ambiguous generic 'suppressed'.
- Drift-skip alerts explicitly bypass the ack gate (they carry the
remediation command and alert once via drift_alerted already).
- Docs: failure-incidents section in the cron guide.
- Tests for the alerted transition + never-resurrect-closed.
---------
Co-authored-by: Laura López Real <113060513+laulopezreal@users.noreply.github.com>
deliver='bot-chat[:<profile>]' is a machine-local pseudo-platform: the
scheduler delivers job output as a real inbound turn in the target
profile's canonical Bot Chat via the chat CLI lane (--in ~ -c "Bot Chat"
--create-if-missing -Q --query-file), the same lane Bot Mode
agent-to-agent messages use. The bot reads the output, acts on it, and
responds in its chat — instead of the output only landing in Run history.
- cron/scheduler.py: token parsing, target resolution (own profile /
named local profile / unknown -> skipped with warning), subprocess
delivery lane with cron.bot_chat_delivery_timeout_seconds (default
600s), preflight exemption, and bot-chat entries in
cron_delivery_targets() for UI pickers. Excluded from 'all' by design.
- tools/cronjob_tools.py: create/update-time validation — named profiles
must exist on this machine (fail at create, not at 3am); deliver schema
documents the new token.
- tui_gateway/methods_tools.py: cron.manage add forwards deliver.
- hermes_cli/profiles.py: list_profile_names() cheap name-only scan.
- hermes-bots plugin: Create Cronjob dialog gains a 'Send results to'
picker (Run history only / <bot>'s chat); bot-chat jobs send the BARE
token on the profile-scoped create so Desktop-side aliases can never
name a profile the backend doesn't have.
- Docs: user cron guide, automate-with-cron, cron-internals.
Machine-local by construction: names resolve only against the executing
machine's ~/.hermes/profiles/, so overlapping profile names across
multiple connected gateways are unambiguous.
A cron job can now pin its own reasoning (thinking) effort, independent
of the global agent.reasoning_effort and per-model reasoning_overrides.
Heavy scheduled analyses can run at high while cheap recurring jobs run
at minimal, without touching the fleet-wide default.
- cron/jobs.py: new optional job field, validated at the storage choke
point against the canonical grammar via the shared
hermes_constants.parse_reasoning_effort (spelling-only; capability
clamping stays owned by the provider transports at send time, same as
config-set effort). Empty string clears on update; invalid values
raise ValueError before anything persists. Not a drift-guard axis.
- cron/scheduler.py: _resolve_job_reasoning_config resolves per-job pin
> agent.reasoning_overrides > agent.reasoning_effort at fire time,
after the auth-fallback model swap (the pin is model-independent by
design). A stored value that no longer parses warns and falls back to
config resolution instead of killing the tick.
- tools/cronjob_tools.py: reasoning_effort on BOTH mutation verbs
(create and update), conditional key in _format_job, schema documents
grammar/precedence/transport clamping/clear semantics. Agent-settable,
unlike model/provider pins: it cannot redirect spend to a different
model.
- hermes cron create/edit --reasoning-effort (empty string clears).
- Docs: cron feature page tip + CLI reference rows.
Tests: tests/cron/test_cron_reasoning_effort.py (32) — store contract,
scheduler precedence incl. byte-identical absent-field behavior and
garbage fallback, tool create/update/clear/error paths, schema surface.
CLI parity for the continuity toggle:
- subcommands/cron.py: --continuity on create; --continuity / --no-continuity
tri-state pair on edit (same store_const pattern as --no-agent/--agent)
- cron.py: forwarded to the cronjob tool; created/edited job summaries print
a "Continuity: on" line
- cronjob_tools._format_job: reports continuity as an explicit boolean and
strips the reserved 'self' entry from the reported context_from list
- cron-job.ts: form reader accepts both shapes (raw store record with 'self'
inside context_from, or formatted record with the explicit flag)
- docs: CLI flag examples in the continuity section
E2E (real argparse -> cron_create/cron_edit -> jobs.json in temp HERMES_HOME):
create --continuity stores ['self']; edit --no-continuity clears; edit
--continuity restores; default-off unchanged. 91 cron/tool tests + 16 CLI
cron tests + vitest 10/10 pass.
- cron/notepad.py: SQLite-backed cron_notepad(job_id, key, value,
updated_at) store in its own profile-local db (cron/notepad.db),
following the executions.py connection/transaction pattern. APIs:
set_note/get_note/delete_note/list_notes/clear_notepad +
render_notepad_section. Documented size caps: 16KB per value,
128-char keys, 64KB per job total; oversized writes raise ValueError.
- cron/scheduler.py: inject non-empty notepads into the job prompt at
the context_from data-injection seam as a clearly-labeled
"Job notepad (persistent across runs)" section that also documents
the CLI write path. Empty notepad renders "" — byte-stable prompts
for jobs that never use the feature.
- hermes_cli/cron.py + hermes_cli/subcommands/cron.py:
`hermes cron notepad <job_id> [get|set|delete|list]` under the
existing cron subcommand tree (no new top-level command, no new
model tool — the agent writes via terminal + CLI).
- tests/cron/test_notepad.py: CRUD, durability, cap enforcement,
prompt injection, byte-stable empty case, read-failure resilience,
CLI handler + dispatch (TDD; watched fail first).
Inspired by: Amp (Sourcegraph) cron notepad (idea-level, proprietary —
zero code).
Add monitor-mode cron jobs: a cheap monitor source (monitor_script or
monitor_url) runs on every tick BEFORE any agent machinery is built.
Its output is hashed as exact bytes and compared to the hash stored
from the last agent-triggering tick:
- unchanged -> agent run suppressed entirely (no LLM, no delivery);
the tick is recorded as a silent no_change run visible in the
executions ledger doc
- changed -> a MONITOR CHANGE DETECTED block (capped unified diff of
previous vs current output + the new output) is injected into the
prompt via the existing extra_prompt seam, then a normal agent run
- first run -> always runs the agent with a baseline block
- source failure -> delivered as an ERROR alert, never treated as a
change; the stored hash is untouched so recovery to prior output
still suppresses
Implementation:
- cron/monitor.py (new): hash/diff/URL-fetch/state persistence.
monitor_script reuses _run_job_script (same ~/.hermes/scripts/
containment + interpreter rules); monitor_url is a bounded GET
(30s, 256KB, http/https only). Output is exact bytes by design —
scripts should emit stable output (documented).
- cron/jobs.py: additive job fields monitor_script / monitor_url /
monitor_state {last_output_hash, last_changed_at}. JSON job records
need no migration. create-time validation: sources are mutually
exclusive and incompatible with no_agent=True.
- cron/scheduler.py: one tight monitor gate in run_job between the
no_agent short-circuit and the LLM path (outside sibling-lane
regions). State persists in jobs.json + a per-job snapshot file, so
suppression survives scheduler restarts.
- tools/cronjob_tools.py: additive optional monitor_script/monitor_url
params on the cronjob tool (create + update, empty string clears),
path containment validated at the API boundary, surfaced in
_format_job.
- hermes_cli: --monitor-script/--monitor-url on `hermes cron create`
and `hermes cron edit`; `hermes cron list` shows the monitor source
and last-changed time.
Tests (tests/cron/test_monitor_kind.py, TDD): unchanged suppresses,
changed injects diff, first run always runs, hash persists across
module reload (restart), script failure is error-not-change with hash
untouched, create/update validation, tool wiring + path-escape reject.
Inspired by: ChatGPT Work monitor tasks (idea-level, docs-only);
enabler: #80774
Per-job cron inference pins are now user-owned: the agent-facing cronjob
tool schema no longer exposes model/provider/base_url, and the registered
handler ignores them even if a model hallucinates the old parameters.
Users set pins via the dashboard, hermes cron create/edit --model/--provider,
or jobs.json directly — and once set, a pin sticks until the user changes it.
Existing agent-era pins are grandfathered untouched.
New cron.model / cron.model_provider config keys give the cron fleet its
own default model, independent of the chat model. Fire-time resolution:
per-job pin > cron.model > HERMES_MODEL > model.default. An axis covered
by the explicit cron-fleet default is deliberate routing, not drift, so
the #44585 fail-closed guard skips it — switching your chat model with
/model or hermes model no longer breaks unpinned cron fleets.
- tools/cronjob_tools.py: drop model param from agent schema + handler;
remove now-dead _resolve_model_override
- cron/scheduler.py: cron.model/model_provider resolution + per-axis
drift-guard skip
- cron/jobs.py: snapshot resolution mirrors the new precedence
- hermes_cli/subcommands/cron.py + hermes_cli/cron.py: --model/--provider
on hermes cron create/edit
- hermes_cli/config.py: cron.model / cron.model_provider defaults
- docs: cron.md model-resolution tip rewritten
* Revert "fix(cron): scope job execution to its owning profile (#32091 follow-up) (#50993)"
This reverts commit 660e36f097.
* Revert "fix(cron): anchor cron storage at the default root home (not the active profile)"
This reverts commit a5c09fd176.
The #32091 fix moved every profile's cron jobs into one shared root store,
but never wired the execution-scoping half it recommended: a job still ran
under whichever profile's ticker picked it up, not its owning profile. So a
job created under `hermes -p donna` could execute with the root profile's
.env / config.yaml / credentials.
- jobs.py: create_job auto-captures the active profile (explicit profile=
override available) and stores it on the job; resolve_profile_home() maps a
profile name to its HERMES_HOME; legacy jobs backfill to 'default'.
- scheduler.py: run_job applies the job's profile via a scoped HERMES_HOME
override (env var + in-process ContextVar) before any .env/config/script
load, restored in finally. tick() routes profile-mismatched jobs to the
single-worker sequential pool so the env mutation can't race.
- cronjob tool threads profile through (NOT exposed in the model schema, to
avoid cross-profile privilege escalation); hermes cron add gains --profile.
E2E verified against a temp HERMES_HOME with a real profile dir: a root-profile
ticker runs a profile='donna' job with HERMES_HOME=donna during execution and
restores the ticker env afterward.
Fully removes the cron per-job 'profile' arg added in #28124: the
cronjob tool schema field, CLI --profile flags on cron create/edit,
job-record storage/validation, the scheduler's _job_profile_context
wrapper, and the script-runner env override. Sequential-partition
logic reverts to workdir-only.
The context-local HERMES_HOME override in hermes_constants and the
subprocess bridging in tools/environments/local.py are kept — they
now have other consumers (dashboard multi-profile, TUI gateway).
Phase 2 of the god-file decomposition plan. main()'s argparse tree is 179
inline add_parser calls in one 3,297-line function. This establishes the
hermes_cli/subcommands/ package and extracts the first group (cron) as the
proof-of-pattern:
- hermes_cli/subcommands/_shared.py: shared parser helpers (add_accept_hooks_flag),
re-exported from main.py for backwards compat.
- hermes_cli/subcommands/cron.py: build_cron_parser(subparsers, cmd_cron=...).
Handler injected so the module never imports main (cycle avoidance).
- main()'s ~155-line inline cron block becomes one build_cron_parser() call.
Behavior-neutral: 'hermes cron create --help' output is byte-identical to
origin/main. main() 3297 -> 3143 LOC.
Validation: tests/hermes_cli/ 6466 passed / 0 failed under per-file process
isolation; new test_subcommands_cron.py covers subactions, aliases, options,
no-agent tristate, injected dispatch, and --accept-hooks.
Victor Kyriazakos