tick() advances a recurring job's next_run_at BEFORE dispatch so a crash
mid-run cannot re-fire it on every restart (at-most-once). That leaves a
window — advance persisted, fire claim not yet taken — in which the process
dying (interpreter already finalizing, executor refusing new futures, SIGKILL,
the Desktop idle-exit from #107485) loses the occurrence silently: the
restarted scan sees only the future next_run_at, writes no execution row and
no log line, and a daily job skips a day.
Contract (documented in cron-internals.md): every recurring occurrence is
accounted for — it runs once, or its skip is logged with a reason.
- The due scan stamps `pending_slot = {scheduled_at, at, by}` in the same save
that records `last_dispatch`; claim_job_for_fire and mark_job_run clear it,
and an explicit schedule / next_run_at / enabled / state rewrite (edit,
pause, resume, run-now) drops it.
- A later scan that finds the stamp with a provably gone owner (this process
and the job not in its running set, or another owner past the fire-claim
lease / dead pid) restores scheduled_at as next_run_at ONCE and logs a
WARNING (cron/occurrences.py::unclaimed_pending_slot). The restored instant
then meets the ordinary policy — completed_occurrence() blocks a second fire
of a slot that already ran, the grace window classifies it late, past-grace
collapses the backlog into one run, cron.catch_up_missed: false skips it
with a logged reason. Never a replay of N slots.
Same store fields on both topologies: a standalone `hermes -p X gateway run`
and a profile served by the default multiplexer evaluate the identical record.
Capture the exact UTC scheduled instant before either due scanning or the
external fire claim advances jobs.json. Bind it to the durable attempt
before worker handoff; manual and unclassified direct attempts stay null.
Consult any retained completed matching row, independently of stale stamps,
claim-time windows, and newer failed attempts. Preserve unknown and legacy
attempt eligibility rather than guessing that a side effect completed.
Real isolated restart probes reproduce duplicate script writes on base and
suppress them on the fix for builtin tick and provider fire. Distinct and
manual occurrences still execute. The campaign-serialized cron suite is
queued; this progressive commit preserves the verified integration step.
Credit holny's issue #104790 and guard proposal #104323; exact identity
replaces the approximation rather than importing its legacy heuristic.
Co-authored-by: holny <holny@foxmail.com>