Files
hermes-agent/agent/prompt_cache_scope.py
T
joaomarcos d7995bffaf fix(cache): qualify the declared key with the conversation generation
The declared key is a per-CHAT identifier and outlives the conversation it
names: reset_session() mints a fresh physical id on /new but keeps the key, and
the idle/daily/suspended policy resets do the same. Hashing the key alone
therefore mapped the conversation before a reset and the one after it onto one
gwk_ scope -- the lifecycle violation @cervantesh raised on #97158 and
@kshitijk4poor reproduced on #97709.

No counter is introduced. The generation that must rotate is already durable:
every one of those boundaries closes the outgoing row with an
_RESET_END_REASONS end_reason, so SessionDB.latest_conversation_boundary reads
the most recent one and declared_conversation_scope hashes 'key|generation'.

That makes the carrier stable across a host's per-response physical ids -- a
host that never resets writes no boundary, so every reply hashes the same value
-- while rotating on every conversation replacement, /new and the policy
auto-resets alike. ended_at only moves forward, so a retired generation can
never be reused: no ABA.

It also cannot drift from the rest of the codebase's notion of a conversation
boundary, because find_latest_gateway_session_for_peer fences on the same set.

The read is on the memoized resolution path, not per API call, and both lookups
fail closed: an unqualified key would span a /new, so a DB error degrades to the
physical-id scope. A SessionDB without the lookup keeps the previous behaviour.

Refs #96811
2026-09-01 02:14:35 -07:00

264 lines
12 KiB
Python

"""Rotation-stable logical cache scope for prompt_cache_key derivation.
Context-compression rotation (legacy ``compression.in_place: false`` mode)
mints a new physical ``session_id`` mid-conversation to segment the
transcript. The prompt-cache scope introduced by #79161 was derived from that
physical id, so every rotation moved the conversation into a fresh cache
bucket even though it is logically the same conversation continuing
(issue #79017).
``resolve_prompt_cache_scope()`` maps the physical session id to the ROOT of
its *compression lineage* — the pre-rotation session id — using
``SessionDB.get_compression_lineage()``, whose fork-aware semantics
(hardened in #79193) give exactly the scope boundaries the cache key needs.
NOT ``SessionDB.get_conversation_root`` / ``run_agent._conversation_root_id``
(the Portal-attribution walk): that one follows ``parent_session_id`` blindly,
collapsing /branch children and whole delegate trees into one id, which would
violate the #79161 isolation this scope must preserve. The two resolvers are
intentionally different — do not "deduplicate" them.
- compression-rotation children walk back to the original segment
(rotation-stable scope — the fix);
- ``/new`` starts a lineage-less session (fresh scope);
- ``/branch`` children (``_branched_from``), delegate subagents
(``_delegate_from``), and tool-tagged children (``source="tool"``) are
explicit fork children and keep their own isolated scope, preserving the
sibling/subagent isolation #79161 established;
- cron fires keep their physical ``cron_<job>_<ts>`` id here — the per-fire
timestamp is stripped later by ``_cache_scope_from_session_id`` exactly as
before.
A host that mints one physical ``session_id`` per RESPONSE (Hermes Studio's
group chat, and ``POST /v1/responses`` with client-managed history, which
mints ``str(uuid4())`` per request) re-keys every conversation-affinity hint
Hermes sends — ``prompt_cache_key`` on both OpenAI-wire transports, plus the
OpenRouter/Nous sticky ``session_id`` and xAI's ``x-grok-conv-id`` through
``portal_tags`` (issue #96811). Those rows carry no lineage, so the walk
above correctly returns the physical id and the scope moves every reply.
Hermes must not infer the logical conversation from the id's SYNTAX (that
rule collides independent client-supplied ids and merges Studio members
truncated past its 96-character boundary — the #79017 failure class). The
host has to declare it, and one carrier already means exactly that:
``gateway_session_key`` — the "stable per-chat key" (``agent:main:telegram:
dm:123``) built by ``gateway.session.build_session_key`` from the
``X-Hermes-Session-Key`` header, which branching deliberately does NOT key
off. ``declared_conversation_scope()`` consumes it, and it wins over the
lineage walk because it is stable across rotation AND across per-response
ids. Two boundaries it must not cross:
- explicit fork children (``/branch``, delegate subagents, tool children)
share their parent's chat key but are separate conversations — the row's
fork markers keep them on their own scope (#79161);
- background-review forks run on a clone of the live runtime, so they are
excluded by ``_persist_disabled`` for the same reason.
The declared key is hashed into ``gwk_<sha256[:24]>`` before it becomes a
scope: unlike a session id it embeds platform/chat/user identifiers, and
this value leaves the process verbatim as OpenRouter's sticky ``session_id``
and xAI's ``x-grok-conv-id``.
The resolution is memoized per (agent, session_id): the lineage walk runs
once per transcript segment — NOT per API call — and re-runs only when
rotation actually changes ``agent.session_id`` (per the no-DB-on-the-hot-path
constraint recorded on #79017). Default installs compact in place and never
rotate, so they hit the memo forever and behave byte-identically to before.
"""
import hashlib
import logging
from typing import Any, Optional
logger = logging.getLogger(__name__)
_MEMO_ATTR = "_prompt_cache_scope_memo"
# Namespace for a scope resolved from a host-declared conversation key.
_DECLARED_SCOPE_PREFIX = "gwk_"
def _lineage_root(session_id: str, session_db: Any) -> Optional[str]:
"""Return the compression-lineage root of *session_id*, or None.
Defensive about the DB handle: test doubles and partially constructed
agents can hand back non-list results — anything that is not a non-empty
list/tuple whose first element is a non-empty string is ignored.
"""
if session_db is None:
return None
try:
lineage = session_db.get_compression_lineage(session_id)
except Exception:
logger.debug("prompt-cache scope lineage walk failed", exc_info=True)
return None
if isinstance(lineage, (list, tuple)) and lineage:
root = lineage[0]
if isinstance(root, str) and root:
return root
return None
def _conversation_generation(session_key: str, session_db: Any) -> str:
"""Return the generation marker for *session_key*'s current conversation.
The declared key is a per-CHAT identifier and deliberately outlives any
single conversation on it — ``reset_session()`` mints a fresh physical id
on ``/new`` but keeps the key, and the idle/daily/suspended policy resets
do the same. Hashing the key alone would therefore map the conversation
before a reset and the one after it onto ONE affinity scope, violating the
#79017/#86733 contract (warm across compression rotation, cold across a
new conversation).
The generation that must rotate is already durable: every one of those
boundaries closes the outgoing row with an ``_RESET_END_REASONS``
end_reason, and ``SessionDB.latest_conversation_boundary`` reports the most
recent one. Qualifying the key with it gives a carrier that is
- stable across a host's per-response physical ids (no boundary is written
when nothing was reset, so every reply hashes the same value), and
- rotating on every conversation replacement, ``/new`` and the policy
auto-resets alike, monotonically — ``ended_at`` only moves forward, so a
retired generation can never be reused.
No counter is introduced anywhere: the marker is read from state the
reset paths already write, and it is read on the memoized resolution path,
not per API call.
Returns ``""`` when the key has never been reset, when the DB does not
expose the lookup, or when it reports nothing.
"""
reader = getattr(session_db, "latest_conversation_boundary", None)
if not callable(reader):
return ""
boundary = reader(session_key)
if boundary is None:
return ""
# Fixed-point so the carrier is byte-identical across repr differences.
return f"{float(boundary):.6f}"
def declared_conversation_scope(agent: Any) -> Optional[str]:
"""Return the host-declared logical conversation scope, or None.
Resolved from ``agent._gateway_session_key`` (the ``X-Hermes-Session-Key``
/``build_session_key`` per-chat key) qualified by the conversation
generation currently live on it (:func:`_conversation_generation`), hashed
together into ``gwk_<sha256[:24]>`` so no platform/chat/user identifier
reaches a provider on the wire and the value stays inside every caller's
length/charset budget.
The key alone would outlive the conversation — it survives ``/new`` and the
idle/daily policy resets by design — so the generation is what makes this
carrier legal: stable across a host's per-response physical ids, and cold
on every conversation replacement.
None — meaning "fall back to the physical-id scope" — when no key was
declared, when this agent is a background-review fork (``_persist_disabled``:
it clones the live runtime, including the key), when the session row is an
explicit fork child (``/branch``, delegate, tool), and on any DB error
during either lookup.
"""
key = str(getattr(agent, "_gateway_session_key", "") or "").strip()
if not key:
return None
if getattr(agent, "_persist_disabled", False):
return None
sid = str(getattr(agent, "session_id", None) or "")
db = getattr(agent, "_session_db", None)
generation = ""
if sid and db is not None:
try:
if db.is_explicit_fork_child(sid):
return None
except Exception:
# Degrade to the physical-id scope rather than risk merging a
# fork onto its parent's key on a transient DB failure.
logger.debug("declared-scope fork check failed", exc_info=True)
return None
if db is not None:
try:
generation = _conversation_generation(key, db)
except Exception:
# Same fail-closed rule as the fork check: an unqualified key
# spans /new, so degrade to the physical-id scope instead.
logger.debug("declared-scope generation read failed", exc_info=True)
return None
carrier = f"{key}|{generation}" if generation else key
digest = hashlib.sha256(carrier.encode("utf-8", errors="replace")).hexdigest()[:24]
return f"{_DECLARED_SCOPE_PREFIX}{digest}"
def resolve_prompt_cache_scope(agent: Any) -> str:
"""Resolve the rotation-stable cache-scope id for *agent*'s conversation.
Returns the host-declared conversation scope when one applies
(``declared_conversation_scope``), else the compression-lineage ROOT of
``agent.session_id`` (the physical id itself when the session has no
compression ancestry, no DB is attached, or the walk fails). The result is memoized on the agent
keyed by the current session id, so the DB walk happens once per
transcript segment rather than once per API call.
"""
sid = str(getattr(agent, "session_id", None) or "")
if not sid:
return ""
db = getattr(agent, "_session_db", None)
# Memo key includes DB presence: an agent that starts DB-less and gains a
# handle later (run_agent._get_session_db_for_recall lazily attaches one)
# must re-resolve instead of staying pinned to the physical id.
key = (sid, db is not None)
memo = getattr(agent, _MEMO_ATTR, None)
if isinstance(memo, tuple) and len(memo) == 2 and memo[0] == key:
return memo[1]
# A declared conversation key outranks the lineage walk: it is stable
# across compression rotation AND across a host's per-response ids, which
# the walk cannot see (#96811).
root = declared_conversation_scope(agent) or (
_lineage_root(sid, db) if db is not None else None
)
scope = root or sid
# Memoize on a successful walk, or when there is no DB to consult at all,
# or when the agent will never persist a row (background-review forks set
# _persist_disabled but still hold a DB handle — without this, every API
# call would re-run the lineage query forever).
# A failed/empty walk on a persisting agent is NOT memoized: falling back
# to the physical id is the correct degraded answer right now (row not
# persisted yet, transient DB error), but pinning it for the whole segment
# would keep the scope wrong after the session row lands.
if (
root is not None
or db is None
or getattr(agent, "_persist_disabled", False)
):
try:
setattr(agent, _MEMO_ATTR, (key, scope))
except Exception:
# Frozen/slotted test doubles — resolution still works, just
# unmemoized.
pass
return scope
def declared_conversation_scope_safe(agent: Any) -> Optional[str]:
"""Never-raising variant of :func:`declared_conversation_scope`."""
try:
return declared_conversation_scope(agent)
except Exception:
logger.debug("declared conversation scope resolution failed", exc_info=True)
return None
def resolve_prompt_cache_scope_safe(agent: Any) -> Optional[str]:
"""Never-raising variant of :func:`resolve_prompt_cache_scope`.
Returns None on any failure (or when there is no scope). Consumers treat
None/empty as "fall back to the physical session_id", so a resolution
failure degrades to pre-#79017 behavior instead of blocking the caller —
important at turn_context's call site, where an exception raised inside
the ``set_runtime_main(...)`` argument list would otherwise skip the whole
runtime binding, not just the cache scope.
"""
try:
return resolve_prompt_cache_scope(agent) or None
except Exception:
logger.debug("prompt-cache scope resolution failed", exc_info=True)
return None