A WebUI/TUI session whose last turn died mid-tool-loop (stale-timeout kill, interrupt, or process restart before the tool result was written) persists a dangling assistant(tool_calls) or interrupted assistant->tool tail. The messaging gateway already strips these tails before replay (the #49201 fix), but the TUI/WebUI resume path fed db.get_messages_as_conversation() straight in as the agent's conversation_history with no cleanup. The model re-issued the unanswered call on every resume -- including after a full WebUI + Gateway restart, since the poison lives in the SessionDB, not memory -- leaving the session permanently 'thinking'. Only deleting the session recovered it. - Extract the two strippers + helper from gateway/run.py into a shared agent/replay_cleanup.py (sanitize_replay_history wraps both). - gateway/run.py re-exports under the historical private names; messaging behavior unchanged. - Both TUI cold-resume sites now sanitize the model-fed history while leaving the display transcript untouched, so the user still sees their full history. Verified E2E against a real SessionDB: dangling and interrupted tails are stripped from the model feed, healthy mid-progress tool sequences are preserved, and the display transcript is always the full raw history.
This commit is contained in:
Teknium
@@ -0,0 +1,140 @@
|
||||
"""Replay-history sanitization shared across resume code paths.
|
||||
|
||||
When a session's last turn dies mid-tool-loop — the process is killed by a
|
||||
restart/shutdown command, a stale-timeout fires, or an interrupt lands before
|
||||
the tool result is written — the persisted transcript can end with a dangling
|
||||
``assistant(tool_calls)`` (no matching ``tool`` answer) or an interrupted
|
||||
``assistant→tool`` block. On resume the model sees that broken tail and
|
||||
re-issues the unanswered call, producing an endless "thinking"/reboot loop
|
||||
(#49201, #29086).
|
||||
|
||||
These pure helpers strip those tails before the history is replayed to the
|
||||
model. They were originally local to ``gateway/run.py`` (which fixed the
|
||||
messaging-gateway path) and are extracted here so every resume surface — the
|
||||
messaging gateway AND the TUI/WebUI gateway — shares the same cleanup instead
|
||||
of the WebUI path silently skipping it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, Dict, List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def is_interrupted_tool_result(content: Any) -> bool:
|
||||
"""Return True if a tool result indicates the tool was interrupted."""
|
||||
if not isinstance(content, str):
|
||||
return False
|
||||
lowered = content.lower()
|
||||
if "[command interrupted]" in lowered:
|
||||
return True
|
||||
if "exit_code" in lowered and ("130" in lowered or "-1" in lowered):
|
||||
return "interrupt" in lowered
|
||||
return False
|
||||
|
||||
|
||||
def strip_interrupted_tool_tails(
|
||||
agent_history: List[Dict[str, Any]],
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Strip interrupted assistant→tool sequences from replay history.
|
||||
|
||||
Older interrupted gateway turns can be followed by a queued real user
|
||||
message, so the interrupted assistant/tool block is not necessarily the
|
||||
final tail by the time we rebuild replay history. Remove any contiguous
|
||||
assistant(tool_calls) + tool-result block that contains an interrupted tool
|
||||
result, while preserving successful tool-call sequences intact.
|
||||
"""
|
||||
if not agent_history:
|
||||
return agent_history
|
||||
|
||||
cleaned: List[Dict[str, Any]] = []
|
||||
i = 0
|
||||
n = len(agent_history)
|
||||
while i < n:
|
||||
msg = agent_history[i]
|
||||
if msg.get("role") == "assistant" and "tool_calls" in msg:
|
||||
j = i + 1
|
||||
tool_results: List[Dict[str, Any]] = []
|
||||
while j < n and agent_history[j].get("role") == "tool":
|
||||
tool_results.append(agent_history[j])
|
||||
j += 1
|
||||
if tool_results and any(
|
||||
is_interrupted_tool_result(m.get("content", ""))
|
||||
for m in tool_results
|
||||
):
|
||||
logger.debug(
|
||||
"Stripping interrupted assistant→tool replay block "
|
||||
"(indices %d–%d, tool_results=%d)",
|
||||
i, j - 1, len(tool_results),
|
||||
)
|
||||
i = j
|
||||
continue
|
||||
if msg.get("role") == "tool" and is_interrupted_tool_result(msg.get("content", "")):
|
||||
logger.debug("Stripping orphan interrupted tool result from replay history")
|
||||
i += 1
|
||||
continue
|
||||
cleaned.append(msg)
|
||||
i += 1
|
||||
|
||||
return cleaned
|
||||
|
||||
|
||||
def strip_dangling_tool_call_tail(
|
||||
agent_history: List[Dict[str, Any]],
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Strip a trailing ``assistant(tool_calls)`` block left with NO answers.
|
||||
|
||||
When a tool call itself kills the gateway process (``docker restart``,
|
||||
``systemctl restart``, ``kill``, ``hermes gateway restart``), the process
|
||||
is terminated by SIGKILL *mid-call* — before the tool result is ever
|
||||
written and before the orderly shutdown rewind
|
||||
(``_drop_trailing_empty_response_scaffolding``) can run. The last thing
|
||||
persisted is the ``assistant`` message that issued the ``tool_calls``,
|
||||
with zero matching ``tool`` rows.
|
||||
|
||||
On resume the model sees an unanswered tool call at the tail and naturally
|
||||
re-issues it — which restarts the gateway again, producing the infinite
|
||||
reboot loop in #49201. ``strip_interrupted_tool_tails`` does not catch
|
||||
this because there is no tool result to inspect for an interrupt marker.
|
||||
|
||||
This strips that dangling tail at the source so there is nothing for the
|
||||
model to re-execute. It only acts when the tail is an
|
||||
``assistant(tool_calls)`` whose calls have NO corresponding ``tool``
|
||||
results — a completed assistant→tool pair (any tool answers present) is
|
||||
left untouched so genuine mid-progress tool loops still resume.
|
||||
"""
|
||||
if not agent_history:
|
||||
return agent_history
|
||||
|
||||
last = agent_history[-1]
|
||||
if not (
|
||||
isinstance(last, dict)
|
||||
and last.get("role") == "assistant"
|
||||
and last.get("tool_calls")
|
||||
):
|
||||
return agent_history
|
||||
|
||||
logger.debug(
|
||||
"Stripping dangling unanswered assistant(tool_calls) tail "
|
||||
"(%d call(s)) — process likely killed mid-tool-call by a "
|
||||
"restart/shutdown command (#49201)",
|
||||
len(last.get("tool_calls") or []),
|
||||
)
|
||||
return agent_history[:-1]
|
||||
|
||||
|
||||
def sanitize_replay_history(
|
||||
agent_history: List[Dict[str, Any]],
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Apply both replay-tail strippers in the canonical order.
|
||||
|
||||
Convenience entry point for resume code paths: removes interrupted
|
||||
assistant→tool blocks anywhere in the history, then removes a dangling
|
||||
unanswered ``assistant(tool_calls)`` tail. Returns the same list object
|
||||
when there is nothing to strip.
|
||||
"""
|
||||
if not agent_history:
|
||||
return agent_history
|
||||
return strip_dangling_tool_call_tail(strip_interrupted_tool_tails(agent_history))
|
||||
+9
-100
@@ -958,106 +958,15 @@ _AUTO_APPEND_MEDIA_TOOL_NAMES = {
|
||||
|
||||
# ---- helpers: detect interrupted tool tails & auto-continue noise ----------
|
||||
|
||||
def _is_interrupted_tool_result(content: Any) -> bool:
|
||||
"""Return True if a tool result indicates the tool was interrupted."""
|
||||
if not isinstance(content, str):
|
||||
return False
|
||||
lowered = content.lower()
|
||||
if "[command interrupted]" in lowered:
|
||||
return True
|
||||
if "exit_code" in lowered and ("130" in lowered or "-1" in lowered):
|
||||
return "interrupt" in lowered
|
||||
return False
|
||||
|
||||
|
||||
def _strip_interrupted_tool_tails(
|
||||
agent_history: List[Dict[str, Any]],
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Strip interrupted assistant→tool sequences from replay history.
|
||||
|
||||
Older interrupted gateway turns can be followed by a queued real user
|
||||
message, so the interrupted assistant/tool block is not necessarily the
|
||||
final tail by the time we rebuild replay history. Remove any contiguous
|
||||
assistant(tool_calls) + tool-result block that contains an interrupted tool
|
||||
result, while preserving successful tool-call sequences intact.
|
||||
"""
|
||||
if not agent_history:
|
||||
return agent_history
|
||||
|
||||
cleaned: List[Dict[str, Any]] = []
|
||||
i = 0
|
||||
n = len(agent_history)
|
||||
while i < n:
|
||||
msg = agent_history[i]
|
||||
if msg.get("role") == "assistant" and "tool_calls" in msg:
|
||||
j = i + 1
|
||||
tool_results: List[Dict[str, Any]] = []
|
||||
while j < n and agent_history[j].get("role") == "tool":
|
||||
tool_results.append(agent_history[j])
|
||||
j += 1
|
||||
if tool_results and any(
|
||||
_is_interrupted_tool_result(m.get("content", ""))
|
||||
for m in tool_results
|
||||
):
|
||||
logger.debug(
|
||||
"Stripping interrupted assistant→tool replay block "
|
||||
"(indices %d–%d, tool_results=%d)",
|
||||
i, j - 1, len(tool_results),
|
||||
)
|
||||
i = j
|
||||
continue
|
||||
if msg.get("role") == "tool" and _is_interrupted_tool_result(msg.get("content", "")):
|
||||
logger.debug("Stripping orphan interrupted tool result from replay history")
|
||||
i += 1
|
||||
continue
|
||||
cleaned.append(msg)
|
||||
i += 1
|
||||
|
||||
return cleaned
|
||||
|
||||
|
||||
def _strip_dangling_tool_call_tail(
|
||||
agent_history: List[Dict[str, Any]],
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Strip a trailing ``assistant(tool_calls)`` block left with NO answers.
|
||||
|
||||
When a tool call itself kills the gateway process (``docker restart``,
|
||||
``systemctl restart``, ``kill``, ``hermes gateway restart``), the process
|
||||
is terminated by SIGKILL *mid-call* — before the tool result is ever
|
||||
written and before the orderly shutdown rewind
|
||||
(``_drop_trailing_empty_response_scaffolding``) can run. The last thing
|
||||
persisted is the ``assistant`` message that issued the ``tool_calls``,
|
||||
with zero matching ``tool`` rows.
|
||||
|
||||
On resume the model sees an unanswered tool call at the tail and naturally
|
||||
re-issues it — which restarts the gateway again, producing the infinite
|
||||
reboot loop in #49201. ``_strip_interrupted_tool_tails`` does not catch
|
||||
this because there is no tool result to inspect for an interrupt marker.
|
||||
|
||||
This strips that dangling tail at the source so there is nothing for the
|
||||
model to re-execute. It only acts when the tail is an
|
||||
``assistant(tool_calls)`` whose calls have NO corresponding ``tool``
|
||||
results — a completed assistant→tool pair (any tool answers present) is
|
||||
left untouched so genuine mid-progress tool loops still resume.
|
||||
"""
|
||||
if not agent_history:
|
||||
return agent_history
|
||||
|
||||
last = agent_history[-1]
|
||||
if not (
|
||||
isinstance(last, dict)
|
||||
and last.get("role") == "assistant"
|
||||
and last.get("tool_calls")
|
||||
):
|
||||
return agent_history
|
||||
|
||||
logger.debug(
|
||||
"Stripping dangling unanswered assistant(tool_calls) tail "
|
||||
"(%d call(s)) — process likely killed mid-tool-call by a "
|
||||
"restart/shutdown command (#49201)",
|
||||
len(last.get("tool_calls") or []),
|
||||
)
|
||||
return agent_history[:-1]
|
||||
# Replay-tail sanitization lives in agent/replay_cleanup.py so every resume
|
||||
# surface (this messaging gateway AND the TUI/WebUI gateway) shares one
|
||||
# implementation. Re-exported under the historical private names so existing
|
||||
# call sites and tests keep working.
|
||||
from agent.replay_cleanup import ( # noqa: E402
|
||||
is_interrupted_tool_result as _is_interrupted_tool_result,
|
||||
strip_interrupted_tool_tails as _strip_interrupted_tool_tails,
|
||||
strip_dangling_tool_call_tail as _strip_dangling_tool_call_tail,
|
||||
)
|
||||
|
||||
|
||||
_AUTO_CONTINUE_NOTE_PREFIX = "[System note: Your previous turn"
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
"""Tests for agent.replay_cleanup — shared replay-tail sanitizers.
|
||||
|
||||
These functions were extracted from gateway/run.py so every resume surface
|
||||
(messaging gateway AND TUI/WebUI gateway) strips poisoned tool-call tails the
|
||||
same way. Regression coverage for #29086 (WebUI session permanently stuck
|
||||
because the dangling tool-call tail was replayed on every resume).
|
||||
"""
|
||||
|
||||
from agent.replay_cleanup import (
|
||||
is_interrupted_tool_result,
|
||||
strip_dangling_tool_call_tail,
|
||||
strip_interrupted_tool_tails,
|
||||
sanitize_replay_history,
|
||||
)
|
||||
|
||||
|
||||
def _user(text):
|
||||
return {"role": "user", "content": text}
|
||||
|
||||
|
||||
def _assistant_tc(name):
|
||||
return {
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{"id": "c1", "type": "function", "function": {"name": name, "arguments": "{}"}}
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _tool(content):
|
||||
return {"role": "tool", "tool_call_id": "c1", "content": content}
|
||||
|
||||
|
||||
def test_is_interrupted_tool_result_markers():
|
||||
assert is_interrupted_tool_result("[Command interrupted]")
|
||||
assert is_interrupted_tool_result("foo\nexit_code: 130 (interrupt)\nbar")
|
||||
assert not is_interrupted_tool_result("exit_code: 0\nclean output")
|
||||
assert not is_interrupted_tool_result("ordinary tool output")
|
||||
assert not is_interrupted_tool_result(None)
|
||||
|
||||
|
||||
def test_strip_dangling_tool_call_tail_removes_unanswered_tail():
|
||||
history = [_user("hi"), _assistant_tc("write_file")]
|
||||
out = strip_dangling_tool_call_tail(history)
|
||||
assert out == [_user("hi")]
|
||||
|
||||
|
||||
def test_strip_dangling_tool_call_tail_preserves_answered_pair():
|
||||
history = [_user("hi"), _assistant_tc("read_file"), _tool("contents")]
|
||||
out = strip_dangling_tool_call_tail(history)
|
||||
assert out == history # answered -> untouched
|
||||
|
||||
|
||||
def test_strip_interrupted_tool_tails_removes_interrupted_block():
|
||||
history = [_user("hi"), _assistant_tc("terminal"), _tool("[Command interrupted]")]
|
||||
out = strip_interrupted_tool_tails(history)
|
||||
assert out == [_user("hi")]
|
||||
|
||||
|
||||
def test_strip_interrupted_tool_tails_preserves_successful_block():
|
||||
history = [_user("hi"), _assistant_tc("read_file"), _tool("ok"),
|
||||
{"role": "assistant", "content": "done"}]
|
||||
out = strip_interrupted_tool_tails(history)
|
||||
assert out == history
|
||||
|
||||
|
||||
def test_strip_interrupted_tool_tails_removes_orphan_interrupted_tool():
|
||||
history = [_user("hi"), _tool("[Command interrupted] exit_code: 130 interrupt")]
|
||||
out = strip_interrupted_tool_tails(history)
|
||||
assert out == [_user("hi")]
|
||||
|
||||
|
||||
def test_sanitize_replay_history_combines_both():
|
||||
# interrupted block in the middle + dangling tail at the end
|
||||
history = [
|
||||
_user("first"),
|
||||
_assistant_tc("terminal"), _tool("[Command interrupted]"),
|
||||
_user("second"),
|
||||
_assistant_tc("write_file"), # dangling
|
||||
]
|
||||
out = sanitize_replay_history(history)
|
||||
assert out == [_user("first"), _user("second")]
|
||||
|
||||
|
||||
def test_sanitize_replay_history_noop_on_clean_history():
|
||||
history = [_user("hi"), {"role": "assistant", "content": "hello"}]
|
||||
assert sanitize_replay_history(history) == history
|
||||
|
||||
|
||||
def test_sanitize_replay_history_empty():
|
||||
assert sanitize_replay_history([]) == []
|
||||
+17
-4
@@ -26,6 +26,7 @@ from hermes_constants import (
|
||||
from hermes_cli.env_loader import load_hermes_dotenv
|
||||
from utils import is_truthy_value
|
||||
from tools.environments.local import hermes_subprocess_env
|
||||
from agent.replay_cleanup import sanitize_replay_history
|
||||
from tui_gateway import git_probe
|
||||
from tui_gateway.transport import (
|
||||
StdioTransport,
|
||||
@@ -5381,13 +5382,17 @@ def _(rid, params: dict) -> dict:
|
||||
_enable_gateway_prompts()
|
||||
try:
|
||||
db.reopen_session(target)
|
||||
history = db.get_messages_as_conversation(target)
|
||||
raw_history = db.get_messages_as_conversation(target)
|
||||
display_history = db.get_messages_as_conversation(target, include_ancestors=True)
|
||||
except Exception as e:
|
||||
if lease is not None:
|
||||
lease.release()
|
||||
return _err(rid, 5000, f"resume failed: {e}")
|
||||
prefix = display_history[: max(0, len(display_history) - len(history))]
|
||||
# Display keeps the full transcript; the model-fed history drops a
|
||||
# dangling/interrupted tool-call tail so a session killed mid-loop does
|
||||
# not replay the unanswered call forever (#29086).
|
||||
prefix = display_history[: max(0, len(display_history) - len(raw_history))]
|
||||
history = sanitize_replay_history(raw_history)
|
||||
# Restore the model/provider/reasoning/tier this chat last used so the
|
||||
# deferred build (and the info below) match the eager path — without them
|
||||
# the build drops the provider ("No LLM provider configured").
|
||||
@@ -5448,13 +5453,21 @@ def _(rid, params: dict) -> dict:
|
||||
)
|
||||
try:
|
||||
db.reopen_session(target)
|
||||
history = db.get_messages_as_conversation(target)
|
||||
raw_history = db.get_messages_as_conversation(target)
|
||||
display_history = db.get_messages_as_conversation(
|
||||
target, include_ancestors=True
|
||||
)
|
||||
# The display transcript keeps every row so the user still sees their
|
||||
# full history. The model-fed history is sanitized: a session whose
|
||||
# last turn died mid-tool-loop persists a dangling assistant(tool_calls)
|
||||
# (or interrupted assistant→tool) tail; replaying it makes the model
|
||||
# re-issue the unanswered call forever — the permanent-"thinking" stuck
|
||||
# session in #29086. The messaging gateway already strips this; this is
|
||||
# the WebUI/TUI resume path picking up the same cleanup.
|
||||
display_history_prefix = display_history[
|
||||
: max(0, len(display_history) - len(history))
|
||||
: max(0, len(display_history) - len(raw_history))
|
||||
]
|
||||
history = sanitize_replay_history(raw_history)
|
||||
messages = _history_to_messages(display_history)
|
||||
tokens = _set_session_context(target)
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user