TUI server shutdown stamps ended_at/end_reason='tui_shutdown' on sessions
whose agent keeps running; every rotation then aborts at
publish_compression_child's liveness check forever (the #88197 wedge; the
amplification half was fixed by #88411).
Class fix: is_automatic_end_reason() in hermes_state_common owns the
"accidental infrastructure cleanup vs deliberate boundary" taxonomy.
publish_compression_child clears automatic stamps in its own transaction
and proceeds (parent re-closes with its TRUE boundary,
end_reason='compression'); the #88411 pre-flush guard no longer aborts on
stamps the publish can heal. Deliberate boundaries (compression,
session_reset, explicit close) still fail closed at both sites.
TEST REPIN (deliberate contract change):
test_ended_parent_aborts_before_the_prepublish_flush pinned
"tui_shutdown stamp => rotation aborts and parent must not grow" — the
abort it required IS the #88197 wedge. Repinned as two tests:
- test_automatic_stamp_no_longer_wedges_rotation: automatic stamp =>
rotation COMMITS (no abort loop, so no growth-by-abort is possible);
- test_deliberately_ended_parent_aborts_before_the_prepublish_flush:
session_reset (deliberate boundary) => still aborts BEFORE the #47202
flush, preserving #88411's no-growth contract where an abort remains
correct.
The class invariant "no aborted rotation grows the parent" holds
everywhere: automatic stamps no longer produce aborts, deliberate
boundaries still abort pre-flush.
When context-compression rotation fires mid-turn, the current user
message was persisted twice into the child session. Root cause: dedup
used id()-seeded sets of copies instead of markers on the live objects.
Replace with _DB_PERSISTED_MARKER-based dedup as the sole authority:
- _ensure_compressed_has_user_turn returns CompressedUserTurnOutcome
- After publish_compression_child succeeds, stamp the live anchor-source
row (not a drifted index) with _DB_PERSISTED_MARKER
- _sync_persisted_markers mirrors stamps from result to live lists by
scoped identity (handles direct-path, adoption divergence, _session_messages)
- Remove _flushed_db_message_ids from rotation commit path (markers replace it)
- Unconditional (loud) imports — no silent fallback
Salvage of #94996 by @fedosis, rebased on top of #95433 (stall-fallback,
already merged). Both conversation_compression.py and run_agent.py are
built from origin/main + #94996's diff applied on top, preserving the
force_terminal refactor and _publish_new_fence from #95433.
Credit: @fedosis original PR #94996.
The rotation path flushes its un-persisted transcript to the parent (#47202)
and only then calls publish_compression_child. The abort handler rolls back
the in-memory transcript and keeps agent.session_id on the parent - its own
comment says "keep the parent live and discard the stale compacted snapshot" -
but the rows the flush just wrote are not part of what it discards. Every
failed rotation therefore leaves the parent transcript longer than it found
it, whatever the failure was.
That is survivable for a one-off failure and pathological for a sticky one.
A parent row carrying ended_at fails the publish on every attempt and nothing
in this path clears it, so each auto-compaction appends another copy of the
current turn to the transcript it was supposed to shrink. Worse, the growth
then satisfies conversation_compression's own len(durable_parent) >
len(messages) check, so the next attempt adopts the inflated snapshot as if it
were genuine concurrent activity and the in-memory transcript doubles too.
Check that one precondition before writing. It is a plain read of the row the
publish is about to read anyway, and it raises the publish's own message, so
split_status=aborted, failure_class=session_split_failed and the rollback path
are all unchanged; a live parent reaches the flush exactly as before.
Deliberately not extended to the compression lease, which is re-acquirable - a
transient miss there would abort a rotation that would otherwise have
committed. old_session_id moves above the flush so a failure raised from here
takes the same in-memory rollback as any other pre-publish failure.
Scope: this fixes the amplification for every abort cause. It does not fix
what marks a live session as ended in the first place (#88197 Bug 1), which
needs a maintainer decision on end-reason taxonomy and is tracked on the
issue; an affected session still aborts every attempt, it just stops making
itself larger while it does.
Refs #88197
The gateway rotation guard (#83339) only protects the rotate path, but
in-place compaction commits inside compress_context() via
archive_and_compact — before the gateway can inspect the result. Add the
anti-growth check at the commit site so both paths are covered: a
compression whose rough output exceeds its input is a strict no-op
(original transcript kept durable, session identity untouched).
Covers the observed failure where session hygiene persisted 426 -> 426
messages and ~379K -> ~688K tokens.
Address rewinds/edits via SQLite messages.id (truncate_before_row_id)
instead of shifting user ordinals. Resolve against in-memory stamps,
then durable session history when live turns drop _row_id; refuse
unknown durable targets with 4018 (no ordinal fallback) and 4030 on
ordinal/row_id mismatch. Stamp _row_id on insert, load row ids on
resume paths, send rowId from Desktop, filter renderer-synthetic ids,
and stop silently resending failed targeted edits without truncation.
Add production-shaped SessionDB tests for resolve and fail-closed paths.
Fixes#82959
The compression heartbeat's terminal 'context compression completed'
stamp force-persists against the PARENT session id (agent.session_id at
stamp time). After the out-of-place rotation the parent is archived but
kept advertising a fresh last_activity_at + terminal label forever.
Clear the parent row's activity labels best-effort after a committed
rotation (keeps last_activity_at so idle clocks stay continuous; the
child carries live labels). Regression asserts the archived parent's
labels are cleared while the child's lineage is intact
(sabotage-verified).
The anti-thrash guard (_ineffective_compression_count) was in-memory
only: a fresh compressor bound to a resumed, already-compacted session
started with compression_count=0 and a disarmed guard, so a
near-threshold session could legally re-compact once per process
restart, forever.
Persist the counter through the durable session-state channel,
mirroring the failure-cooldown (#54465) and fallback-streak (af7dceaf7)
pattern:
- hermes_state.py: sessions.compression_ineffective_count column
(declarative reconciliation adds it on existing DBs) +
get/set_compression_ineffective_count accessors.
- context_compressor.py: every strike/clear verdict routes through
_record_ineffective_compression_verdict() which writes through to the
session row (no-change verdicts skip the DB write);
bind_session_state() loads the persisted value; the compression
rotation boundary carries the counter onto the child row;
update_model()'s reset also clears the durable copy; the
ineffective-only fast path in _automatic_compression_blocked() is
removed because the counter is now durable and another agent's clear
must unblock a stale local snapshot.
- conversation_compression.py: _refresh_persisted_compression_guards
re-reads the counter alongside cooldown + fallback streak.
Reset semantics are unchanged: any real provider reading below the
threshold still clears the counter — and now clears it durably too.
Resolves the residual gap identified in #54923 by @lanyusea (the
second-threshold mechanism was superseded by persisting the existing
guard state).
Co-authored-by: lanyusea <lanyusea@gmail.com>
Covers the follow-up hardening: continuation-marker and summary-as-user
tails keep the flagged standalone snapshot (zero-user provenance #69292
verified via _transcript_has_real_user_turn on the projected rows),
stale snapshot rows are refreshed in place, a previously merged snapshot
is stripped before re-injection, and an all-completed todo store injects
nothing (#26981).
After context compression, the preserved todo list was unconditionally
appended as a standalone user message. When the compressed transcript
already ends with a user message (common case), this creates consecutive
user/user turns — a role-alternation violation some providers reject.
Fix: fold the snapshot into the trailing user message (blank-line separated)
when one exists with plain-string content. Falls back to append when the
tail is non-user, empty, or has structured (list) content.
Rebased on current upstream/main.
Closes#53890
Follow-ups on top of #64731's cwd/git_repo_root inheritance:
- git_branch joins the parent-row backfill (same NULL-only COALESCE hop):
the Desktop sidebar branch chip otherwise vanishes at every compaction
boundary even though the workspace didn't change.
- Belt-and-suspenders for #59527: compression forks (parent already ended
with end_reason='compression') also inherit the gateway origin columns
(user_id/session_key/chat_id/chat_type/thread_id/display_name/
origin_json) at DB-level child creation. The gateway re-records the peer
after rotation (d5b4879d4), but a hard crash in the window between child
creation and that write left the child unrecoverable by
find_latest_gateway_session_for_peer. Scoped to compression forks only —
delegate/subagent children (parent still live) must NOT inherit routing
keys, or peer recovery could repoint gateway traffic into a subagent's
session.
- Behavioral test driving the real _compress_context rotation path,
asserting the child row carries cwd/git_repo_root/git_branch and the
origin columns.
The ineffective-compression counter is not durable; when it is the sole
reason the gate is blocked there is nothing in the DB that could unblock
it, so re-reading the guard rows on every gate check for the rest of the
session is pure waste. Guard test pins the no-DB-touch behavior.
Follow-up on the salvaged #64511 commit:
- _automatic_compression_blocked() now refreshes durable guard state
(cooldown + fallback streak) when — and only when — the in-memory
snapshot says blocked, then re-evaluates. The should_compress()
pre-gates (preflight/turn paths) consult this before ever reaching
compress_context, and a stale fallback streak has no expiry timer, so
without a gate-level refresh a cleared durable row could never unblock
a prebound agent. The unblocked hot path pays no DB reads.
- A refresh that finds no durable cooldown row no longer clears a live
local cooldown whose DB persist FAILED (_cooldown_persist_failed):
an empty row is not evidence another agent cleared it, and honouring
it would reopen the #11529 thrash window. A successful durable
round-trip (record or read) makes the DB authoritative again.
- Guard tests for both directions (red on the pre-fix code), including
a hot-path test asserting the unblocked gate never touches the DB.
These 7 test sites assert rotation behavior (fork, child sessions, lock
contention, logging session-context follows id rotation, boundary hooks fire
on rotation). Pin each builder to in_place=False explicitly so they keep
exercising the retained rotation fallback regardless of the global default
(flipped to True in #38763). Rotation stays a working opt-out fallback and
deserves continued coverage — these are NOT deleted.
Pinned sites:
- test_compression_concurrent_fork._build_agent_with_db
- test_compression_logging_session_context._build_agent_with_db
- test_compression_rotation_state._build_agent_with_db
- test_compression_boundary_hook._make_agent (2 helpers: CompressionBoundaryHook + SessionCompressEvent)
- test_compression_concurrent_sessions._build_agent_with_db
Three state-loss bugs at the compression rotation boundary, fixed together
because they all live in the same ~80-line rotation block:
- #33618: a persistent /goal did not follow the rotation. load_goal does a
flat per-session lookup with no lineage walk, so a goal silently died when
compression minted a fresh child id. Added migrate_goal_to_session() and
call it after the child session is created (move-not-copy: the parent row
is archived as cleared so exactly one active goal row exists).
- #33906/#33907: if the child create_session raised (FK constraint,
contended write), the outer handler only warned and let the agent continue
on the NEW id — which has no row in state.db — producing an orphan session.
Now the rotation rolls agent.session_id back to the still-indexed parent
(reopening it) instead of stranding the conversation on a phantom id.
- #27633: the compaction-boundary on_session_start notification omitted the
platform kwarg, so context-engine plugins saw source=unknown for every
message after the boundary. Forward platform (matching the initial
session-start call in agent_init.py).
Co-authored-by: denisqq <21260182+denisqq@users.noreply.github.com>
Co-authored-by: zccyman <16263913+zccyman@users.noreply.github.com>
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Teknium