The orphan reaper had two gaps that let agent-browser daemons accumulate
indefinitely inside a single long-lived hermes process:
1. `_reap_orphaned_browser_sessions()` ran exactly once, before the cleanup
loop started, so a leak appearing after boot could never be recovered.
2. `owner_alive is True` skipped unconditionally. In-memory session tracking
is lost on any exception path between spawn and registration, but the
owner PID stays up — so such a daemon was skipped forever.
The daemon-side `AGENT_BROWSER_IDLE_TIMEOUT_MS` is not a backstop for (2):
it does not fire when the daemon itself is wedged, e.g. after Chrome's
framework was replaced underneath it by an auto-update.
Observed on macOS: five agent-browser daemons (96 Chrome processes) built up
over 10 days inside an 18-day-uptime hermes process, holding roughly 5 CPU
cores busy and driving the load average past 100. Four of those processes
were still running a Chrome framework version that had since been replaced
on disk, spinning at ~85% CPU each.
Changes:
- Re-run the reaper every `BROWSER_ORPHAN_REAP_INTERVAL` (300s) from inside
the cleanup loop. Cycle 0 preserves the existing startup reap.
- When the owner is alive but the session is untracked, fall back to idle
age: reap past `BROWSER_ORPHAN_GRACE_SECONDS`, defined as
`max(1h, 20 x inactivity_timeout)`. Unknown age fails safe.
- Add `_socket_dir_idle_seconds()` — the newest mtime under a session's
socket dir. Every browser command writes `_stdout_<cmd>` / `_stderr_<cmd>`
there, making it a last-activity marker that survives hermes restarts and
does not depend on in-memory bookkeeping surviving an exception path. It
scans directory entries rather than reading the directory mtime alone:
command names repeat, and rewriting an existing `_stdout_click` updates
that file's mtime but not the directory's, so a dir-mtime-only check would
report a busy session as idle and reap it.
Sessions still present in `_active_sessions` are never touched at any age,
and the new path still goes through `_verify_reapable_browser_daemon`, so
the anti-spoof / anti-PID-recycle guarantees from #14073 are unchanged.
Adds 9 tests: idle-age unit tests (including the dir-mtime regression),
spared/reaped/fail-safe cases for a live owner, the identity-guard gate on
the new path, and a periodic-reap test asserting more than one reap per
cleanup-thread lifetime.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
andy
c99a45b28e