cf60ebbdfd264c3fc067b9e3230df8752c50d2ec
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Teknium
|
3c7069bdcb |
refactor(gateway): dead-code removal, helper unification, defensive-layer collapse and rationale-preserving comment compaction across 40 modules
authz_mixin, browser_control_broker, delivery, delivery_ledger, display_config, drain_control, hosted_room_links/peer/policy_checkpoint, hosted_rooms, platform_registry, relay/__init__, relay/ws_transport, run.py and slash_commands.py (comments), session_context, session_state, streaming_tts_consumer, turn_lease and small modules. - HostedRoomPolicyCheckpoint._apply_event -> per-kind handler table - WebSocketRelayTransport._handle_frame -> frame-handler table - GatewayAuthorizationMixin: unified adapter setting/flag/extra readers - dead symbols removed (verified zero references): RoomLinkProbe/select_room_link, relay_bot_username, is_restart_loop_tripped, debug_rows, DeadTargetRegistry.all_dead, BrowserControlBroker.detach_owner/_prune_tickets, StreamingTTSConsumer.started/_enqueue_done/ _iter_stream_chunks/_next_stream_chunk, RecoverableHandleCache.status_for, _auth_env, _copy_default_catalog, _parse_timestamp_prefix, _present_* helpers, _send_result_error_kind, _truthy_env, SessionFieldView/TurnLeaseTokenView dunder shims, and their orphaned tests. - lost WHY/invariant text from the earlier compaction restored compactly (541 hunks audited) |
||
|
fangliquanflq
|
39d2d858fd | fix(gateway): persist hygiene failure cooldown rung | ||
|
kshitij
|
aaf9688519 |
refactor(gateway): extract the hygiene recovery gate and forward the failure reason
Follow-up to
|
||
|
kshitij
|
c0d974b19f |
fix(gateway): escalate the session-hygiene compaction cooldown on repeat failures
A gateway session whose summary model keeps timing out no longer retries compaction on the same fixed interval forever. The in-agent compressor already escalates repeat summary timeouts 60 -> 300 -> 900s (ContextCompressor.record_timeout_failure), but that ladder reads the in-memory _consecutive_timeout_failures counter and bind_session_state() zeroes it (context_compressor.py:1645). Session hygiene constructs a FRESH AIAgent for every run (gateway/run.py:16820) and re-binds state each time, so from the gateway that streak is structurally always 0 -- only the flat hygiene_failure_cooldown_seconds (300s) could ever be recorded. Issue #79624 reported exactly that steady state: an oversized session (1053 messages, ~119.5k tokens) whose aux model always timed out, re-attempting compaction every 300s across five days until the reporter deleted the session by hand. Track the streak on PersistentState instead, which outlives the per-run agent and is not cleared by turn/boundary resets, so consecutive hygiene failures climb 300 -> 900 -> 2700s and then saturate. Both failure sites (progress timeout and aborted compression) feed it; a real compression resets it, so a session that recovers starts from the first rung again. The ladder multiplies the configured base, so operators who tuned hygiene_failure_cooldown_seconds keep their first rung. Per-session, so one wedged chat cannot penalize other conversations. Deliberately NOT changed, since each is a maintainer policy call rather than a defect (all three are written up on #79624): - no durable failure-streak column, so escalation still resets on restart - the gateway 30s / in-agent 120s / aux-client 300s-floor timeout mismatch - no `hermes doctor` check or `hermes sessions list` marker for a session stuck in a compression-failure cooldown Note the reported exit(1) is NOT a crash: it is the deliberate _signal_initiated_shutdown path (gateway/run.py:26746-26751, #5646) that lets systemd Restart=on-failure revive the gateway after a bare SIGTERM, and it fires on every `systemctl restart` independently of compaction. The compaction log lines appear after the shutdown line because the gateway-owned executor is torn down with shutdown(wait=False, cancel_futures=True) (run.py:21164), so an in-flight turn keeps logging during teardown. Full analysis on the issue. Post-review hardening (Phase 2c + /simplify-code found five real defects in the first cut): - the recovery gate hand-rolled `_new_tokens < _approx_tokens` when a canonical predicate already existed: `compression_made_progress` (agent/turn_context.py, #39548). They disagree on 3 of 5 cases -- the hand-rolled form misses a row-count win when the summary keeps the token estimate flat, misses one where the summary is slightly MORE verbose (so a genuinely recovered session would keep escalating forever), and counts a sub-5% wobble as recovery. Now reuses the shared predicate, promoted from `_compression_made_progress` to a public name with the old private name kept as a back-compat alias so the existing importer (tests/agent/test_protected_tail_pressure_61932.py) and any patcher of that symbol keep working. - the reset was gated on "not aborted", but the degenerate "did not rotate or compact in place" branch (#21301) is NOT aborted and yields zero reduction, so a session wedged there reset its streak every run and could never escalate -- silently defeating the fix. Now gated on real progress. - no absolute ceiling: base * 9 reaches 9h at an operator base of 3600s, indistinguishable from "compaction switched off". Added _HYGIENE_COOLDOWN_MAX_SECONDS = 3600, mirroring the in-file _RECONNECT_BACKOFF_CAP precedent. - the reset used the get-or-create accessor to write a 0 that was already 0, materialising a _sessions entry (never evicted). Now peeks. - the abort verdict was probed twice, leaving the reset/record mutual exclusion implicit; a future await between the probes would have broken it silently. Computed once into _hyg_aborted. Tests: 19 new in tests/gateway/test_hygiene_failure_cooldown_ladder.py -- ladder escalation, saturation, the absolute cap, per-session isolation, reset-on-recovery, custom/zero base, PersistentState scoping (a mutation moving the field to TurnState fails), degraded runners, the progress gate, the exact progress-predicate semantics the gate depends on, and end-to-end that the escalated value is what reaches the state DB. All 12 mutations caught, including ones that restore the flat cooldown (the original bug), ungate the reset, swap the canonical predicate back for the hand-rolled comparison, remove the cap, and share the streak globally; the harness hard-errors when a mutation cannot be applied, since a silently no-op mutation check is worse than none -- an earlier version of it WAS silently no-opping after a refactor. The gate's contract test slices by AST node span rather than a fixed character count, which had already truncated once as the block grew. gateway hygiene + session-state + the three touched agent compression suites: 50 passed; ruff clean. E2E with real imports demonstrates the premise rather than asserting it: bind_session_state zeroes the in-agent counter, and the recorded deadlines go 300 -> 900 -> 2700 -> 2700 -> 2700s where they were previously a flat 300s. Reported by @yucezerey (#79624), whose state.db column dump and "deleting the session fixed it" datapoint made the real mechanism findable. |
||
|
teknium1
|
ab08e8fc76 |
refactor(gateway): consolidate 19 session-keyed dicts into SessionState (turn/conversation/persistent scopes; eliminates wholesale-reset races)
GatewayRunner carried ~19 separate Dict[str, ...] attributes keyed by
session_key, each with an ad-hoc lifecycle. They now live in one
`self._sessions: dict[str, SessionState]` (gateway/session_state.py) with
three lifecycle scopes and a `_session_state(key)` get-or-create accessor.
Mechanical refactor: same state, same semantics, new container.
Migration table (dict -> old decl line -> current clear path -> new home):
| legacy dict | decl | cleared by (before) | SessionState field |
|------------------------------------------|-------|--------------------------------------------------|----------------------------------------|
| _running_agents | 3505 | _release_running_agent_state; stop() .clear() | turn.agent |
| _running_agents_ts | 3506 | same | turn.started_ts |
| _active_session_leases | 3507 | same (+ lease.release()) | turn.lease |
| _busy_ack_ts | 3539 | same | turn.busy_ack_ts |
| _turn_lease_tokens ((key, gen)-keyed) | 3518 | _release_turn_lease (generation-guarded) | turn.lease_token + turn.lease_generation |
| _session_model_overrides | 3585 | _CONVERSATION_SCOPED_STATE funnel | conversation.model_override |
| _pending_one_turn_model_restores | 3586 | funnel; one-shot pop in turn finally | conversation.one_turn_restore |
| _session_reasoning_overrides | 3589 | funnel; lazy-init dict swap ~5692 (RACE) | conversation.reasoning_override |
| _session_service_tier_overrides | 3592 | funnel; lazy-init dict swap ~5738 (RACE) | conversation.service_tier_override |
| _last_resolved_model ("*" = process-wide)| 3527 | funnel | conversation.last_resolved_model |
| _queued_events | 3537 | funnel; lazy-init dict swap ~5204 (RACE) | conversation.queued_events |
| _pending_turn_sidecar_notes | 3597 | funnel; lazy-init dict swap ~19832 (RACE) | conversation.sidecar_notes |
| _session_ephemeral_pin | 3601 | agent-cache evict pop; lazy swap ~19885 (RACE) | conversation.ephemeral_pin |
| _session_vc_last | 3604 | agent-cache evict pop; lazy swap ~19864 (RACE) | conversation.vc_last |
| _pending_approvals | 3611 | boundary security funnel; stop() .clear() | persistent.approvals |
| _update_prompt_pending | 3623 | security funnel; update watcher pops | persistent.update_prompt_pending |
| _pending_native_image_paths_by_session | 3538 | one-shot consume; lazy swap ~12944 (RACE) | persistent.native_image_paths |
| _pending_messages (runner-level, str) | 3519 | _interrupt_and_clear_session pop; stop() flush | persistent.pending_command_text |
| _session_run_generation | 3540 | NEVER (monotonic, #28686) | persistent.run_generation (never reset)|
Races eliminated: every `self._X = {}` lazy-init/reset replaced the WHOLE
dict, so a writer on session A racing a lazy init triggered by session B
could lose its entry. All six such sites (_session_reasoning_overrides
~5692, _session_service_tier_overrides ~5738, _queued_events ~5204,
_pending_turn_sidecar_notes ~19832, _session_ephemeral_pin ~19885,
_session_vc_last ~19864, _pending_native_image_paths_by_session ~12944,
_turn_lease_tokens ~13644) are now per-session field writes on an existing
SessionState; a reset can no longer cross sessions structurally.
Registry successors:
- _release_running_agent_state -> state.turn.clear() (one structured reset
instead of the drifting pop-list; still pops the slot lease and calls
lease.release() first; still generation-guarded).
- _CONVERSATION_SCOPED_STATE funnel -> state.conversation.clear(); the
tuple is retained for legacy plain-dict stores not yet folded in
(_pending_model_notes) and for the public test contract.
- _turn_lease_tokens' (key, generation) tuple key -> lease_token +
lease_generation fields; release/rebind only match when the generation is
current, preserving the #28686/#64934 ownership check.
- _session_run_generation stays monotonic on persistent.run_generation and
is never cleared (conversation boundaries and turn releases don't touch it).
Compatibility adapters: tests (and a few mixin call sites) access the old
dict names directly (137 direct assignments to _running_agents alone), so
each legacy name is kept as a thin @property returning a live MutableMapping
view over the corresponding SessionState field (legacy_dict_property /
legacy_lease_token_property in session_state.py). Setter accepts a plain
dict (the `runner._X = {...}` test pattern); views support ==, in, len,
.get/.pop/.clear. The shutdown path in _stop_impl deliberately keeps
duck-typed legacy-attribute access because test fakes borrow it with plain
dicts.
Name collision noted (NOT touched, out of scope): gateway/platforms/base.py
has its own _pending_messages Dict[str, MessageEvent] (adapter-level slot);
the runner-level Dict[str, str] of the same name is what moved to
persistent.pending_command_text.
Entry leaks preserved (follow-up, no new eviction in this PR): SessionState
entries in self._sessions are never evicted, matching the old dicts — e.g.
_last_resolved_model, _session_run_generation, _session_vc_last entries for
dead sessions leaked before and their fields still occupy a SessionState now.
Verification: 123 tests/gateway files referencing the old names +
_release_running_agent_state + _CONVERSATION_SCOPED_STATE all pass (sole
failure test_feishu.py::test_websocket_sdk_accepts_channel_ua_tag is
pre-existing, stash-verified); 8 non-gateway test files touching the names
pass (266 tests); `import gateway.run` subprocess smoke OK; ruff clean;
post-migration grep shows zero non-comment `self._<oldname>` references in
run.py outside the property adapters and the duck-typed shutdown block.
|