set_session_metadata() and advance_compression_session()'s repoint both
stamped entry.updated_at = now. updated_at is the user-activity clock that
drives idle/daily reset policy and the restart-resume freshness gate
(suspend_recently_active, #85709), so a background metadata write (e.g.
Slack thread watermark) or a background compression repoint on a long-idle
session could make it look freshly active and get it falsely resume_pending
after a gateway restart.
These are the last internal stamp sites after 784f733cf (recover) and
5462f689b (touch_activity gating): drop the stamps, keep the durable save.
Follow-up to #85895 (closed) — credit @GodsBoy for the report-side push and
@chelsealong for the analysis on #85709.
Both session recovery paths (the startup stale-entry repoint and the
lazy in-message recovery) rebuilt the routing entry with updated_at=now
and never consulted _should_reset, so an opt-in idle/daily session_reset
policy was silently dead across any gateway restart: a recovered session
always looked freshly active, and since every subsequent message bumps
updated_at, a session recovered stale could then never age out at all.
Fix in three parts:
- _create_entry_from_recovered_row derives updated_at from the durable
last_activity_at the finder already returns on the row (no extra DB
round-trip; the original PR added SessionDB.get_last_activity for
this, unnecessary post-#82633), falling back to created_at. An
invalid or missing started_at now maps to epoch 0 instead of now — an
invalid durable timestamp must look old, never freshly active.
reset_had_activity is set from the row's durable activity/message
signals so the continuity hint stays accurate.
- _recover_session_from_db evaluates _should_reset on the rebuilt entry:
an overdue session is durably promoted to a reset boundary
(promote_to_session_reset, falling back to end_session) and the stale
mapping is dropped instead of repointed.
- _query_recoverable_session no longer reopens the row; the
get_or_create_session recovery phase evaluates _should_reset first and
either feeds the normal auto-reset create path (reset notice,
prev_session_id continuity, durable promotion) or reopens and
publishes the recovered entry exactly as before.
Behavior is unchanged under the default session_reset mode "none":
_should_reset returns None there, so recovery still resumes every
recoverable row — only users who opted into idle/daily resets see the
policy actually applied across restarts.
Cherry-picked from #78618 and adapted to the #82633 finder.
(cherry picked from commit 31c71f762961638c199287fc6ffe836115c4892b)
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
When the #54878 self-healing path drops a stale sessions.json entry, the
fix at gateway/session.py:1765 now checks _should_reset() before falling
through to DB recovery. This test covers the case where the stale entry's
session is overdue under an idle/daily reset policy — it must create a
fresh session, set auto-reset metadata, and NOT call reopen_session().
Regression tests for find_latest_gateway_session_for_peer and
SessionStore stale-routing self-heal when end_reason is ws_orphan_reap.
Pin manual approval mode in blocking E2E tests so smart aux-LLM
resolution does not flake CI.
The routing-heal added to get_or_create_session calls
SessionDB.get_compression_tip; the stale-guard suite's bare MagicMock db
returned a Mock the heal then assigned as session_id, failing JSON
serialization. Model the real contract (a non-compressed session's tip is
itself) so the heal is a correct no-op.
Detect a routing key whose session is already ended in state.db
(end_reason set) inside get_or_create_session and drop the stale entry
instead of silently routing the message into a closed session.
Previously the only runtime cleanup of sessions.json was the startup
_prune_stale_sessions_locked (#52808/#54138), which requires a restart.
A session ended while the gateway stays alive — any path that finalizes
the DB row without clearing sessions.json — left a live routing key
pointing at a closed session. get_or_create_session never consulted
end_reason, so it returned that stale entry and every subsequent message
was silently dropped (no log, no error, no response) until the next
restart. This is the live-gateway variant of #52804/FM9, which needed an
actual gateway crash.
The guard drops the stale entry and falls through to
_recover_session_from_db, which reopens agent_close-ended rows and
resumes the SAME session_id (transcript preserved); if the row ended for
a non-recoverable reason (e.g. /new) it correctly starts a fresh
session. A warning is logged so the event is visible (the field
incident reported zero log output).
Adds tests/gateway/test_session_store_runtime_stale_guard.py covering
the _is_session_ended_in_db helper and the end-to-end routing self-heal
(recover-vs-fresh, live-entry untouched, stale-wins-over-suspended,
force_new short-circuit).
Closes#54878.
Co-authored-by: David Gutowsky <david.gutowsky@gmail.com>
Teknium