devops
9b9cfbc1fd
fix(cron): reconcile stale in-flight claim against executions ledger (t_8b5480b3)
...
The age-only stale-claim sweep (t_3778a491, already on main) force-releases
an in-memory _running_job_ids claim only once it is older than
max(2*interval, 30m). A leaked claim that is YOUNG (inside its allowance)
while the durable executions ledger already proves the last run ended stays
wedged: the job is returned as due every tick, _submit_with_guard short-
circuits on 'already running', and next_run_at keeps fast-forwarding with no
execution — the exact 2026-08-14 recurring-router incident (t_20e23f84),
which survived a gateway restart because the in-memory age bound alone could
not see a run the ledger had already finished.
sweep_stale_inflight now reconciles each in-flight claim against the durable
executions ledger (cron/executions.db): if the job's MOST RECENT execution
row is terminal (completed/failed/unknown), the run provably ended, so the
claim is stale by construction regardless of its in-memory age and is force-
released. This is a persisted-state recovery path: the ledger is written by
the worker that ran the job and read by ANY ticker process (including one
that started AFTER the leak), so a leaked claim is recoverable without
force-run/resume and without depending on which process holds it in memory.
A ledger-terminal release is authoritative — it does not write a synthetic
mark_job_run failure (the ledger already records the outcome).
Added TestLedgerTerminalReconciliation (4 tests): young+terminal -> released
(RED on main, GREEN here), no-ledger-row -> not released, running-row -> not
released, old+terminal -> released once without synthetic failure.
2026-08-17 16:56:54 +05:30
..
2026-08-16 22:07:00 -07:00
2026-08-08 04:19:49 -07:00
2026-08-16 22:10:02 -07:00
2026-08-12 13:30:01 -05:00
2026-08-17 03:03:47 -07:00
2026-08-09 22:09:49 -04:00
2026-07-29 13:39:40 -07:00
2026-08-17 16:56:54 +05:30
2026-07-29 13:39:40 -07:00
2026-08-01 10:52:34 -07:00
2026-07-28 08:25:25 -07:00
2026-08-15 13:35:32 -07:00
2026-08-17 16:07:05 +05:30
2026-08-17 02:56:17 -07:00
2026-08-16 06:38:38 -07:00
2026-08-13 23:43:15 +05:30
2026-08-04 17:36:26 -04:00
2026-08-08 18:11:02 -07:00
2026-08-03 20:35:47 +05:30
2026-08-17 02:56:17 -07:00
2026-08-13 11:11:53 -07:00
2026-08-17 03:25:07 -07:00
2026-08-13 02:38:28 -07:00
2026-08-12 19:13:32 -07:00
2026-08-13 22:09:44 -07:00
2026-08-15 23:37:22 -07:00
2026-08-08 12:33:19 -07:00
2026-08-17 03:03:35 -07:00
2026-08-16 23:26:10 -07:00
2026-08-07 10:11:05 -07:00
2026-07-31 22:33:11 -07:00
2026-08-17 14:16:21 +05:30
2026-08-12 17:25:30 -05:00
2026-08-06 05:00:17 +05:30
2026-08-13 22:44:23 +05:30
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-08-01 15:05:05 +05:30
2026-08-16 01:50:20 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-08-02 00:11:50 -07:00
2026-08-15 23:37:22 -07:00
2026-07-25 07:17:52 -07:00
2026-07-29 13:10:23 -07:00
2026-08-15 00:25:00 -07:00
2026-07-29 15:12:28 -07:00
2026-08-14 22:23:00 -07:00
2026-08-14 15:41:38 -05:00
2026-08-03 17:19:49 -07:00
2026-07-29 13:10:23 -07:00
2026-08-14 21:48:14 -07:00
2026-07-29 13:10:23 -07:00
2026-08-02 00:11:50 -07:00
2026-08-08 05:21:09 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-08-03 20:30:32 +05:30
2026-07-29 13:10:23 -07:00
2026-08-12 19:13:32 -07:00
2026-08-16 06:29:53 -07:00
2026-08-09 22:09:49 -04:00
2026-08-15 00:36:03 -07:00
2026-07-29 13:10:23 -07:00
2026-08-09 22:09:49 -04:00
2026-08-15 23:37:22 -07:00
2026-08-09 22:09:49 -04:00
2026-07-29 13:10:23 -07:00
2026-07-29 18:13:09 -07:00
2026-08-16 19:53:07 -07:00
2026-07-29 18:55:10 -07:00
2026-08-13 23:43:15 +05:30
2026-08-13 23:43:15 +05:30
2026-07-29 17:35:16 -05:00
2026-07-26 19:30:32 -07:00
2026-08-13 02:38:28 -07:00
2026-08-14 21:35:30 -07:00
2026-08-14 22:33:58 -07:00
2026-08-08 19:46:35 -07:00
2026-08-14 21:58:09 -07:00
2026-08-14 21:58:09 -07:00
2026-08-14 21:58:09 -07:00
2026-08-16 11:34:40 -07:00
2026-07-31 22:39:11 -07:00
2026-07-26 16:06:50 -05:00
2026-08-13 02:38:28 -07:00
2026-08-13 18:39:54 +00:00
2026-08-13 18:39:54 +00:00
2026-08-14 21:58:09 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-24 09:49:00 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 18:13:09 -07:00
2026-08-17 02:40:59 -07:00
2026-08-15 01:55:22 -07:00
2026-07-29 13:10:23 -07:00
2026-08-07 19:44:35 +05:30
2026-07-29 13:10:23 -07:00
2026-07-29 21:30:53 -07:00
2026-07-29 19:53:17 -07:00
2026-08-11 17:06:15 -05:00
2026-08-16 23:26:10 -07:00
2026-07-30 00:08:28 -05:00
2026-07-29 13:10:23 -07:00
2026-08-03 17:38:13 +05:30
2026-08-14 21:36:41 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 15:12:28 -07:00
2026-07-31 07:18:02 -07:00
2026-07-31 22:35:16 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-08-09 22:09:49 -04:00
2026-07-29 13:10:23 -07:00
2026-08-14 21:45:08 -07:00
2026-08-14 21:50:48 -07:00
2026-08-13 11:49:14 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 19:48:37 -07:00
2026-08-14 13:51:26 -07:00
2026-08-14 13:51:26 -07:00
2026-08-12 18:55:14 -07:00
2026-08-10 17:02:56 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 21:30:53 -07:00
2026-08-08 12:33:19 -07:00
2026-07-29 13:10:23 -07:00
2026-08-07 00:41:58 -04:00
2026-08-02 10:04:48 -07:00
2026-08-16 20:43:13 -07:00
2026-08-10 17:02:56 -07:00
2026-08-10 17:02:56 -07:00
2026-07-29 13:10:23 -07:00
2026-08-03 20:37:17 +05:30
2026-08-02 21:32:13 +05:30
2026-07-31 23:18:12 -07:00
2026-08-06 07:49:42 -07:00
2026-08-04 11:26:15 +05:30
2026-08-12 19:43:29 -07:00
2026-08-12 19:43:41 -07:00
2026-08-15 02:22:58 -07:00
2026-08-08 14:18:26 +05:30
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-08-15 00:36:03 -07:00
2026-07-29 13:10:23 -07:00
2026-08-13 09:36:02 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-08-07 09:44:21 -07:00
2026-07-29 13:10:23 -07:00
2026-08-16 06:26:37 -07:00
2026-08-06 23:08:23 -05:00
2026-08-16 06:26:37 -07:00
2026-07-29 13:10:23 -07:00
2026-08-06 19:35:47 -06:00
2026-08-09 13:43:14 -07:00
2026-07-28 11:57:37 -07:00
2026-08-12 20:02:07 -07:00
2026-08-02 21:30:54 +05:30
2026-08-02 23:10:28 +05:30
2026-08-14 12:02:05 -04:00
2026-08-13 02:38:28 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-29 13:10:23 -07:00
2026-07-24 23:06:05 -07:00