The 120s bound added in #80912 proceeded WITHOUT the lock on timeout
(fail-open). That degraded mode fires on every overlap with a HEALTHY
long-running workdir job - the write lock is legitimately held for the
holder's entire agent run - and a workdir-less job that proceeds unlocked
executes its shell/file/code commands with the holder's process-global
TERMINAL_CWD override visible: silent wrong-directory execution, the
exact corruption _ReadWriteLock exists to prevent (see
test_reader_never_observes_writer_override). A degraded WRITER was worse:
it clobbered the active holder's override mid-run and later restored a
pre-wait snapshot over the holder's value.
Fail closed instead: on timeout the job errors loudly with an actionable
message (stagger the holder's schedule / drop its workdir) and is retried
on its next tick. A failed job is visible and recoverable; a job that ran
in the wrong directory is neither.
The bound is now derived from the cron inactivity limit
(HERMES_CRON_TIMEOUT, default 600s) + 60s margin instead of a flat 120s:
a wedged holder stops touching its activity clock, so the inactivity
monitor reaps it and releases the lock within that limit - waiters only
fail when even the monitor could not clear the holder. Healthy workdir
jobs shorter than the inactivity limit can no longer fail their waiters.
Design follows @necoweb3's #63959 (fail-closed semantic); its 30s flat
bound would have failed every waiter overlapping a healthy >30s workdir
run, which is why the bound is derived instead.
kshitij
11ce6419c3