Review P1. _profile_home_for_key() returned the same None for three
different states — multiplexing off / legacy agent:main namespace, a named
profile whose directory does not exist yet, and a resolution error — and
_db_for_key() collapsed all of them to the ambient store.
That recreated the very split this change removes. The enrollment bridge
provisions profiles/<name>/ at runtime, so a key such as
agent:fitness:telegram:dm:1 can legitimately be seen first: the first lookup
landed in root state.db, and the next one, after provisioning, in
profiles/fitness/state.db. One qualified session identity, two physical
stores. The resolver-exception path fell open the same way.
Ownership is now tri-state:
- no named owner -> ambient DB (single-profile behavior intact)
- named owner + home -> that profile's DB
- named owner, unresolvable -> None, and a warning; never root
Callers already treat a missing DB as "skip the mutation", which is the
defer-don't-misroute behavior wanted here. _append_transcript_message is the
one path reached with an id the entry-point guard did not check (the
compression-child id), so it now raises explicitly and lets the caller's
retry queue hold the row instead of relying on an AttributeError.
Tests exercise the effect boundary rather than cache state: a named key
before its profile exists leaves root untouched and lands only in the
profile store once provisioned, and a resolver exception fails closed too.
Both fail against the previous two-state behavior by returning a live
SessionDB where None is required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review feedback on the memo introduced with _profile_home_for_key. The
problem is sharper than "no invalidation on profile deletion": caching the
miss pinned a profile that appears AFTER the gateway started to the ambient
store for the life of the process, which is the exact failure this helper
exists to prevent.
That is not hypothetical — an enrollment bridge can provision
profiles/<name>/ at runtime, so a key is legitimately seen before its
directory exists.
Memoize hits only. A miss costs one profile_exists() stat and recurs only
for profiles that genuinely do not exist, so the hot path for real profiles
is still a dict hit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#88734 made SessionStore._db follow the ambient HERMES_HOME so a multiplexed
profile's rows reach its own state.db. That is correct for the inbound message
path, which installs the scope via _profile_runtime_scope. Nothing else does.
_session_expiry_watcher (gateway/run.py) walks the single process-wide
_entries dict — every profile's keys — and finalizes expired sessions with no
scope installed, so _db resolved the ROOT store for rows that live under
profiles/<name>/state.db. The scoped inbound path and the unscoped background
path then maintained two copies of the same logical session whose end_reason
drifted apart independently. Once they disagreed, the #54878 stale-routing
guard read one copy while the routing index pointed at the other, and a live
conversation was dropped and recreated — silently, since that branch only sets
was_auto_reset when a reset policy also fired.
Field evidence from a live two-profile install: session 20260814_234313 was
end_reason=None in the root store but agent_close in the profile store, while
20260822_225807 was inverted. Both directions, which rules out a single
mis-scoped writer.
The owning profile is already encoded in the session key, so derive the store
from it: _profile_home_for_key / _db_for_key, plus _db_for_session_id for the
entry points addressed by session id. 40 self._db uses across 14 methods now
resolve that way. No signature changed and no existing test was modified.
_profile_home_for_key returns None when multiplexing is off, when the key
carries the legacy agent:main namespace, or when the profile has no live
directory, so single-profile installs resolve exactly where they always did.
The explicit-path branch still goes through SessionDB.__init__ ->
_ensure_test_isolation, keeping the live-DB guard over per-profile paths.
Part of #66887. The routing-index half — _routing_scope() and the sessions.json
mirror still pinned to one frozen sessions_dir while the handle moves — is left
for a follow-up rather than mixed in here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the #99758 review P1 (andrexibiza): with a generation claim in
play, `interrupt()` called `_admit_hard_cancel()` BEFORE the claim was
validated at the final mutation edge, and the production
`CompressionCommitFence.cancel_before_commit()` irreversibly sets
`_cancelled = True` whenever no commit has started. So a watchdog abort
that ultimately DECLINED (real progress landed in the window, claim went
stale) had already killed the recovered turn's legitimate pending
compression commit — `begin_commit()` refuses a cancelled fence forever.
Generation authority covered interrupt publication but not the
compression-fence mutation that preceded it.
Split hard-cancel admission into two halves:
- `_wait_for_compression_commit()` runs pre-claim and is NON-mutating:
it only blocks when `commit_in_flight` is true (the started-commit
branch of the production fence waits for `finish_commit` without
cancelling), so the interrupt still publishes only after an in-flight
SessionDB mutation has finished — exactly as before.
- `_cancel_pending_compression_commit()` runs AFTER
`_consume_claim_and_publish_first_state()` survives, so the
destructive pending-commit cancellation can never outlive a stale
claim. If a commit crossed its boundary in between, it is no longer
fence-cancellable and completes on its own.
Regression coverage (both use the real `CompressionCommitFence`):
- `test_declined_abort_does_not_cancel_pending_compression_commit`:
parks the interrupt at the claim-reservation release, lands real
progress (G+1), lets the interrupt decline, then proves
`fence.begin_commit()` still admits. Red on the pre-fix tree
(mutation-checked: the fence was left cancelled).
- `test_declined_abort_parks_and_leaves_fence_operational`: the
in-flight-commit window variant — activity lands while the interrupt
waits on a started commit; the interrupt declines and a fresh
`begin_commit()` still admits afterwards.
- The round-6 witness (`...resumes_inside_interrupt_publication`) now
models an in-flight commit (`commit_in_flight = True`) so its park
point stays inside the pre-claim wait, matching the new admission
shape.
Also updates the `interrupt()` docstring for the deferred destructive
cancellation.
Closes the #95663 round-8 review blocker (false settlement before
commit veto): the pre-commit surface (`_surface_stall`) logged
"Force-aborting the turn and stopping lease renewal" and warned the
user "aborting it so the session can recover" BEFORE `_commit_abort`
could veto — so a turn that resumed during the warning window (or an
exceptional interrupt path that declines fail-closed) was reported as
force-aborted with lease stopped while it actually continued running.
- Split the surface: `_surface_stall` is now observational only ("no
progress for Ns; attempting recovery"), and the definitive
aborted/lease-stopped settlement moves to a new
`_surface_committed_abort` that runs only after `_commit_abort`
succeeds and the turn lease is deactivated.
- Rate-limit repeated pre-commit surfaces per observed generation: a
turn whose aborts keep declining no longer re-logs an ERROR and
re-warns the user every poll interval.
- Add the committed-path regression test
(`test_watchdog_publishes_definitive_settlement_only_after_commit`)
and extend the declined-path witness
(`...resumes_during_warning`) to assert no committed-abort or
definitive pre-commit claim appears when the abort is vetoed. Both
fail on the pre-fix tree (mutation-checked).
- Document the `_interrupt_turn` lease-loss asymmetry (fires
unconditionally, no generation claim — losing the lease means the
process no longer owns the session).
- Trim review-round archaeology from comments/docstrings (keep the
WHY, drop the round numbering), and drop the dead
`cancel_event` compat note from the test fence.
- Document `agent.turn_liveness` in the configuration guide.
On top of PR #95663 by Finn763 (cherry-picked with authorship
preserved).
Add a turn-liveness watchdog keyed to the agent activity clock: a turn
that stalls mid-flight while the durable lease keeps renewing is logged
loudly, surfaced to the UI, force-interrupted, and — when the hard
interrupt cannot unwind the wedge — lease renewal is stopped so
stale-turn cleanup can reclaim the session.
Race safety (rounds 3/4/6 of the #95663 review, all folded into this
squashed commit):
- AIAgent.interrupt(require_generation=G) re-validates the generation
claim at the last instant before the hammer; a stale claim abandons
the abort and the turn continues.
- The claim is reserved under the activity lock, invalidated by any real
progress in _touch_activity(), and consumed immediately before the
first observable interrupt publication; exceptional paths fail closed.
- Claim consumption and the first interrupt publication are atomic
inside one _liveness_activity_lock() critical section; unclaimed
interrupts publish lock-free so AIAgent stand-ins without the liveness
seam keep working (CI 33096454629 regression, fixed here).
Deterministic race regressions (written red-first) in
tests/run_agent/test_turn_liveness_watchdog.py cover the
post-revalidation window, the consume-to-publication window, the
exceptional path, and atomic claim consumption.
Round-7 rebuild: single squashed commit on current origin/main; the
former four-commit lineage (241f8e484..299122558 on merge base
6defe7eb6c) no longer exists, so no surviving commit carries a red
exact-object CI record, and no empty CI-trigger commit was added.
A non-admin '/sessions all' or '/resume --all' silently downgraded to
chat-scoped listing with zero feedback, which reads as 'my session
vanished' (community Telegram report). Both surfaces now append a notice
that cross-chat listing requires a configured admin. Follows up the
salvaged current-session '(current)' marker (PR #68556, fixes#68547):
sibling tests updated to pin the new contract, new i18n key
gateway.resume.all_requires_admin added to all 17 locales, docs updated.
_record_db_file_identity's PRAGMA fallback is a pure read — route it
through _read_ctx() instead of the writer lock (Pattern C gate).
test_codex_turn_persists_each_message_exactly_once leaked a live
SessionDB into shutil.rmtree, racing the WAL sidecars ('Directory not
empty' on CI); close the handle first and rmtree with ignore_errors.
Main removed the live-path runtime FTS rebuild (18ac3c4fb6) and #99652
narrowed the corruption classifier, after the salvaged PR #89364 branched.
Update its tests to assert the current contract: no fail-open detach on a
replaced file (fts_enabled stays True, no stale marker), fail-open (not
rebuild) for genuine same-file FTS corruption, and FTS-provenance errors
for the guard probe since generic malformed no longer reaches fail-open.
Detect same-inode cp via a generation stamp, halt FTS repair, and divert
unwritten transcripts to sessions/<id>.jsonl plus the gateway pending spool.
Co-authored-by: Cursor <cursoragent@cursor.com>
Addresses Enough1122's two non-blocking review notes on PR #92316:
1. Lease vs deadline arithmetic: the state.db init/migration/repair leases
(600-900s) are authoritative against the 300s default deadline by design.
Add explicit comments at all three lease sites pinning the trade-off:
single lease is deliberate (clamped to _MAX_LEASE_S=900), honest worst
case is up to the lease duration of zombie time on a wedged DB phase,
accepted over per-chunk renewal complexity in the migration loops.
2. Arm-site coverage: add a structural contract test asserting every
documented entry point (hermes_cli/main.py argv fast-path,
hermes_cli/gateway.py config-bridge re-arm, gateway/run.py backstop,
cli.py legacy --gateway) actually calls arm_startup_watchdog (or its
aliased import), so a future entry point can't silently ship unwatched.
Addresses the two class-level review blockers on PR #89750:
1. Bounded hard-exit seam (escort thread). The forensic fire path
(logger.critical, dump record, faulthandler, lifecycle ledger) can
itself wedge — the parked main thread may hold the logging handler
lock, or the disk may be full/hung. _fire() now starts an exit-escort
daemon thread BEFORE any forensics; it is free of log handlers,
filesystem access, module loads and application locks, and hard-exits
with the restart code after _FIRE_EXIT_BOUND_S unless the normal fire
path signals completion. Adversarial tests hold the logging handler
lock / hang the dump write at fire time and assert the exit seam is
still reached.
2. Phase-owned progress leases (report_startup_progress). Process CPU
time proves process activity, not startup progress: an unrelated busy
thread could extend forever while startup sits parked (false
negative), and I/O-bound repair/backup accrues ~zero CPU and would be
killed (false positive). Long synchronous startup phases now declare
authoritative, clamped (_MAX_LEASE_S), renewable progress leases:
state.db _init_schema + the version-gated data-migration chain
(hermes_state_schema) and repair_state_db_schema (hermes_state) are
wired. CPU progress remains only as a bounded fallback, capped at
_MAX_CPU_EXTENSIONS, with leases outranking the cap. Adversarial
tests cover both directions (lease saves zero-CPU legitimate work;
capped CPU noise no longer hides a parked deadlock).
Fire-path dump record now includes lease_count/last_lease_phase for
forensics. gateway/startup_watchdog.py shim re-exports
report_startup_progress.
OOF-298
Independent review of the initial startup-liveness watchdog surfaced two
P1s and three P2s. All are addressed here.
P1 — legitimate slow startups (large state.db schema migrations inside
SessionDB.__init__, which run synchronously before the loop starts) could
exceed the fixed 300s deadline and restart-loop. The watchdog now checks
process CPU time (time.process_time(), process-wide) when the deadline
expires: continuous CPU consumption means a live migration, so the deadline
is extended (with a warning log per extension). The OOF-298 deadlock class
parks every thread in futex waits and accrues ~zero CPU, so it still fires
on schedule. Documented limitation: a spinning busy-wait deadlock reads as
progress and won't fire — the observed incident class is parked threads.
P1 — import-time deadlocks were outside coverage. The implementation moved
to a stdlib-only top-level module (hermes_startup_watchdog), and
hermes_cli/main.py arms it via an argv fast-path ("gateway" + "run" in
argv) BEFORE the heavy module-level import graph. gateway/startup_watchdog
remains as a re-export shim so the intuitive import path keeps working for
the disarm site, tests, and REPL use. Import-lightness is a correctness
property, tested via AST inspection: at fire time the wedged main thread
may hold the import lock, so the fire path performs no imports on its own
thread — the lifecycle-ledger write runs on a bounded-join helper thread
and os._exit happens regardless.
P2 — disarm/fire race: the handle now has an explicit state machine
(armed → disarmed | firing) guarded by a lock; whichever transition takes
the lock first wins, so a disarm landing after deadline expiry but before
the fire transition is honored. Regression test forces the exact
interleaving by blocking inside the CPU probe.
P2 — uncovered entry points: cli.py --gateway and scripts/hermes-gateway
run_gateway() now arm the watchdog before importing the gateway graph.
hermes_cli/gateway.py run_gateway() keeps an idempotent backstop arm for
programmatic callers.
P2 — respawn-storm backoff interaction: the storm breaker's intentional
backoff sleep (up to minutes, ~zero CPU — indistinguishable from a parked
deadlock) now calls kick_startup_watchdog(extra_s=backoff) so the deadline
is pushed past the sleep instead of firing mid-backoff.
Also: the faulthandler stack dump is now additionally written to
logs/gateway-startup-watchdog.log (stderr may be absent on detached/
windowless runs); the disarm site in gateway/run.py moved inside the
loop-confirmed branch (if the loop is NOT live, the milestone was not
reached and the watchdog must stay armed); hermes_startup_watchdog added
to pyproject py-modules so sealed venvs ship it; SERVICE_RESTART_EXIT_CODE
is duplicated in the stdlib-only module with a parity test against
gateway.restart.
Tests: 38 in tests/gateway/test_startup_watchdog.py (contracts incl.
stdlib-only AST check and shim re-export identity, config resolution,
arm/disarm/kick, CPU-progress extension vs no-progress fire, probe-failure
fails toward firing, disarm-vs-fire race, dump record + file stacks,
lifecycle ledger, custom exit code).
A hosted gateway (hermes-doubleam-2568) deadlocked at startup with every
thread parked in futex_wait_queue before the asyncio loop came alive:
zero log lines, /health unreachable — but s6 saw a live PID so it never
respawned the process, and a stale gateway_state.json from the previous
life told every status surface "draining" for ~30 hours.
Every existing liveness backstop assumes startup succeeded: the
loop-liveness watchdog is armed inside the running loop's startup path,
the shutdown watchdog arms at stop(), and the heartbeat file is written
by an asyncio task. None can fire when the process wedges before the
loop exists.
New gateway/startup_watchdog.py: a plain daemon OS thread armed at
process entry (both gateway.run.main() and the `hermes gateway run` CLI
wrapper), disarmed the moment GatewayRunner confirms a live event loop —
the point where the existing loop-liveness watchdog takes over. If
startup neither reaches that milestone nor exits within the deadline
(default 300s; slowest legitimate pre-loop work is the 120s-bounded MCP
discovery wait), the watchdog:
* dumps all-thread stacks via faulthandler,
* appends a JSON record to logs/gateway-startup-watchdog.log,
* records the exit in the NS-608 lifecycle ledger
(reason=startup_liveness_watchdog) so the next boot classifies it
instead of reporting an unclean SIGKILL/OOM death,
* os._exit(75) so s6/systemd respawn the process.
Config is env-only (HERMES_STARTUP_WATCHDOG=0 to disable,
HERMES_STARTUP_WATCHDOG_TIMEOUT_S to tune, floor-clamped to 30s):
the watchdog must be armed before config.yaml is loaded — a wedge
during config parsing is exactly in scope — so it cannot depend on
config for its own enablement. Everything is best-effort; a watchdog
failure never affects the startup it observes.
Arm sites are placed after the PID-file/--replace conflict guards so a
--replace loser exiting early never arms a watchdog. Disarm happens even
when the loop guards are config-disabled (gateway.loop_watchdog: false)
— the startup watchdog only covers the pre-loop window, never adapter
connects or steady-state, so WhatsApp pairing / npm cold installs are
unaffected.
Tests: tests/gateway/test_startup_watchdog.py (29 tests — config
resolution, arm/disarm idempotency, fire path with captured exit,
lifecycle-ledger marking, dump record, disable knob).
Fixes OOF-298.
On Windows, the Desktop update preflight scan (hermes_cli._scan_venv_blockers)
classified only python -m http.server as a stoppable blocker. A hermes serve
or dashboard backend that survived the Desktop teardown (or was launched
manually) either dead-ended the hand-off with venv-blocked, or slipped past
it and kept venv\Scripts\hermes.exe mapped, so the updater quarantine
failed with os error 32 (#98336).
The CLI updater downstream already owns exactly this holder class with
positive-identity rungs: _ledger_reapable_backend_pids reaps dead-spawner
orphans and _ledger_manual_serve_holders stops manual serves and relaunches
them on their recorded host/port. Mirror the existing pausable-gateway
exemption: defer ledger-verified serve/dashboard holders to those rungs
instead of reporting them as blockers.
Identity is positive-only, per the #99558 guard contract: token-parsed
subcommand (never substring, #90778) + live-verified (pid, create_time)
ledger entry with matching purpose + provable ownership (spawner dead,
unrecorded, or the hand-off Desktop itself — an ancestor of the scan,
verified by pid+create_time). A backend supervised by any other live
process keeps blocking; ledger unreadable fails closed.
Fixes#98336
Builds the escalation layer on top of HexLab98's heal-log windowing
(salvaged from PR #96916):
- Per-session heal counters (heal events + messages healed) tracked by the
repair path in agent_runtime_helpers.py, session totals preserved across
10-minute log windows.
- Threshold escalation: after N heals in a session window (default 3,
configurable via agent.sanitizer_heal_escalation_threshold in
config.yaml, 0 = off) log ONE ERROR carrying session id + heal pattern
(events/messages/window/threshold), then stay quiet.
- ONE-TIME out-of-band user notice queued at the threshold and delivered by
the conversation loop through _emit_warning (status callback -> gateway
status message / CLI print). Never injected into conversation context or
the wire copy: prompt caching, role alternation, and durable history are
untouched. Never re-arms on a new window; scoped per session.
- Counters visible in diagnostics: get_sanitizer_heal_stats() rendered in
the /debug share // hermes debug report, and the config key surfaced in
hermes dump overrides. errors.log carries the ERROR line for `hermes logs
errors`.
Second leg of the masoria 20-minute 'Summarizing' stall (Aug 31 2026
bundle): after the Codex timeout, compression fell back to OpenRouter,
which defaulted the omitted output cap to the model's full 65,536-token
window and rejected with '402 ... can only afford 7117' — on an account
whose balance easily covered a summary. Three fallbacks, three 402s,
zero summaries.
- _create_with_progress: when a 402 names an affordable budget, retry
ONCE with that cap (minus 64-token margin, 512-token floor); plain
exhaustion 402s and within-budget 402s re-raise unchanged. Single
funnel covers primary, fallback, and retry call sites.
- extends the #41055 OpenRouter max_tokens preservation onto the current
_build_call_kwargs gate (explicit caps survive; None still omitted).
Live A/B against a mock OpenRouter enforcing a 7117-token budget with
the real SDK + real adapter: main fails with the exact bundle 402;
fixed branch retries clamped and returns the summary.
_build_call_kwargs strips max_tokens for non-Anthropic providers to
avoid wire-format issues (Copilot, ZAI, GPT-5). However, OpenRouter
free/limited-credit tiers need max_tokens because the model's full
output window exceeds the credit budget, causing HTTP 402.
Without max_tokens, the 402 triggers fallback to a text-only model
which then fails with 'unknown variant image_url, expected text'.
Include max_tokens when provider is 'openrouter' or base_url contains
'openrouter.ai'.
Fixes#41035
Adapt the #99660 blocked-before-first-event watchdog test to the FD-ownership
contract: the stranger-thread Timer marks the timeout but never close()s;
the owning thread surfaces the TimeoutError and releases the FDs on unwind.
_CodexCompletionsAdapter.create arms a daemon threading.Timer that calls
client.close() when the aux Responses stream exceeds its timeout. On a
stalled stream -- the failure the timeout exists for -- the Timer is the
only thing that fires, so the close runs on a thread that does not own
the in-flight httpx connection.
That is the FD-ownership violation the repo already fixed twice on the
main transport (#29507, #67142, #70773): close() releases the raw TLS fd
while the owner's OpenSSL BIO still caches that integer, the kernel
recycles it into the next open() in the process -- a SessionDB or
kanban.db handle -- and the owner's unwinding TLS flush writes an
application-data record into that database file.
agent/auxiliary_client.py had no thread-ownership machinery at all: the
guarded twins (_retire_shared_openai_client, _abort_request_openai_client)
live in run_agent.py and are unreachable from this adapter, which holds no
AIAgent reference.
Dispatch on ownership the way chat_completion_helpers already does: from a
stranger thread only force_close_tcp_sockets() (shutdown(SHUT_RDWR), which
is FD-safe from any thread), and let the owning thread release the FDs when
it unwinds. The owner-thread caller (_check_cancelled) keeps closing
directly. Cache eviction (#23432) is unchanged.
The previous form was `mock_install.assert_not_called(), ("msg")` — a bare
tuple expression whose parenthetical never surfaces as a failure message.
Switch to `assert mock_install.call_count == 0, "msg"` so the diagnostic
actually appears when the guard regresses (review feedback on #93163).
On Windows, hermes update launched from the install's own venv can never
complete the managed-runtime repair: Windows keeps the image of the
updater's venv\Scripts\python.exe (and the waiting hermes.exe launcher
ancestor) mapped until exit, so the park rename in _cut_over_candidate
always fails with ERROR_ACCESS_DENIED. The pre-flight holder scan
deliberately excludes the calling process and its ancestors, so the guard
passes and the repair burns its retries on a structurally unwinnable
rename - forever, since the failure is non-fatal and the 'next update
will retry' message is misleading for this case.
Detect the self-lock (sys.executable or a launcher ancestor inside the
live venv) before provisioning and defer with actionable guidance instead
of walking into the doomed cutover. The deferral happens pre-provisioning,
so the incomplete generation-* leftovers the reporter observed are no
longer produced. No-op off Windows: POSIX renames work while the updater
maps the tree. Mirrors the existing _defer_update_for_self_lock pattern.
Regression tests prove the fix bites: neutralized guard -> repair proceeds
to provisioning (red); restored guard -> deferred before provisioning
(green). Verified on Windows 11 against a real venv.
Closes#93032
Follow-ups on top of the #94595 cherry-pick, implementing the maintainer
review's two blockers:
Blocker 1 (turn-admission chokepoint): _run_prompt_submit itself now runs
the ownership admission, so synthesized turns that never pass through the
prompt.submit RPC handler (crash auto-continue from cold session.resume,
wake-ups) are fenced too. Auto-continue additionally checks ownership
BEFORE emitting message.start and leaves the marker in place, closing the
#94778 shape where backend B resumed a session backend A was actively
running and auto-continued A's fresh interrupted-turn marker into a
duplicate concurrent turn.
Blocker 2 (fail-closed registry semantics): try_acquire_active_session no
longer converts an unreadable/corrupt registry into an untracked go-ahead.
Ownership uncertainty is a distinct typed refusal —
SESSION_COORDINATION_UNAVAILABLE — because "could not prove ownership" must
never be collapsed into "no owner exists". The TUI gateway claim helper
fails closed on claim exceptions for every surface, not just desktop.
Also: empty session ids short-circuit to a no-op lease (nothing to fence,
and the strict registry schema rejects empty ids), and the existing
fail-open tests were updated to assert the new fail-closed contract.
Cherry-picked from PR #94595 (author: Futahua) onto current main, with the
maintainer-review revision points folded in during the rebase:
- the lease engages UNCONDITIONALLY: try_acquire_active_session no longer
returns a disabled no-op lease when max_concurrent_sessions is unset;
the concurrency cap stays an orthogonal, optional policy checked second
- ownership uncertainty fails CLOSED (SESSION_COORDINATION_UNAVAILABLE)
instead of degrading to an untracked go-ahead: a corrupt/unreadable
registry must not be collapsed into 'no owner exists' (review blocker 2)
- the ownership admission sits at the _run_prompt_submit chokepoint that
EVERY fresh turn source crosses, and crash auto-continue acquires (or
bails) BEFORE emitting message.start — closing the #94778 bypass where
backend B's auto-continue ran a duplicate turn while backend A was live
(review blocker 1)
- the TUI gateway claim helper fails closed on claim exceptions for every
surface, not just desktop
- CLI and messaging-gateway call sites pass live_session_id metadata so
the (pid, live id) re-entrancy identity protects them from self-fencing
on a leaked lease
Co-authored-by: teknium1 <teknium1@users.noreply.github.com>
Salvaged from #98741, composed on top of the merged #98424 preflight
fail-closed boundary. A host-ceiling compression timeout is now a typed,
thread-safe outcome consumed by every automatic caller:
- conversation_compression.py: threading.local + per-agent lock timeout
state (mark/reset/read helpers) upgrading #98424's simple attribute
where overlapping automatic/manual compression entrypoints matter;
the _last_compression_timed_out attribute stays as compat mirror.
- conversation_loop.py: the mid-turn pre-API pass and the provider
overflow (413/400 context_length_exceeded) recovery path end the turn
with the typed compression_exhausted recovery contract instead of
re-sending the unchanged oversized request and re-entering compression
in the same turn.
- run_agent.py/turn_context.py: forwarder resets the typed state per
attempt; the #98424 turn-start check reads it through the typed helper.
Tests: thread-safety/atomicity of the state helpers, overflow-recovery
non-re-entry, and typed terminal result.
When the default/global state.db is corrupt at gateway startup,
SessionStore degrades (_db=None) and record_gateway_session_peer never
self-heals. Under multiplexed profile routes the AIAgent's lazy
_ensure_db_session was then the ONLY durable write for the session, and
it created the row identity-less (session_key/chat_id/chat_type/
thread_id/user_id/origin_json all NULL) — unrecoverable by
find_latest_gateway_session_for_peer, so Telegram chats forgot prior
turns.
Persist the routing identity the agent already carries into
create_session; plain CLI sessions keep the old identity-less shape.
Salvaged leg 1 of #88804; the transcript-recovery leg is covered by the
scope-aware session DB resolution already on main.
Follow-ups on top of the salvaged commits from PR #87111 (@HexLab98) and
PR #87265 (@JoaoMarcos44):
- keep main's #92991 stall watchdog (150s progress-based) as the single
steady-state liveness probe instead of adding a second overlapping one
- orphaned-client aclose() cleanup uses the wall-clock thread deadline and
is tracked in _background_tasks so a wedged close can neither hang nor
leak one task per reconnect attempt (from #87265's review findings)
- merge #87265's no-keepalive getUpdates pool (max_keepalive_connections=0)
with #87111's TCP-keepalive socket options on all transports
- add tests/gateway/test_telegram_closewait_windows_live.py: live probes
against a real half-closing HTTP server, skipif non-win32, wired into
the on-demand windows-venv-e2e lane (wine2e/**)
Prevent the dedicated getUpdates pool from reusing server-closed connections and replace a polling HTTP client left open after a timed-out CLOSE-WAIT drain. Keep the general Bot API pool reusable so concurrent sends and edits are unaffected. Add regression coverage for both transport limits and stale-client replacement. Fixes#87057
The native generateContent adapter never runs uncapped: when
model.max_tokens is unset it sends maxOutputTokens=65,535
(GEMINI_DEFAULT_MAX_OUTPUT_TOKENS) because Gemini treats an omitted cap
as a low internal default. The context compressor's trigger is
pct×(window − max_tokens), and constructing it with max_tokens=None
reserved 0 — so on a 128K Gemma window the trigger landed at 98,304
while the real safe input budget was 65,537, and the provider 400'd
before compaction fired.
Live repro (real imports, temp HERMES_HOME, native Gemini base_url,
window=131072, max_tokens unset):
before: compressor.max_tokens=None, threshold_tokens=98304,
wire maxOutputTokens=65535 → trigger ABOVE the safe budget
after: compressor.max_tokens=65535, threshold_tokens=64000 → below it
Scoped to the native Gemini wiring (provider names + native base_url via
is_native_gemini_base_url; the /openai compat endpoint is excluded). The
generic provider-default reservation gap remains tracked in #63839.
Reported by @Artemonim in #57275 (residual claim 4).
model.ollama_num_ctx is resolved AFTER the context compressor is
constructed, so a config that sets only ollama_num_ctx (without
model.context_length) ran every request at the smaller served num_ctx
while the compressor still targeted the probed GGUF window (e.g. 256K
Gemma metadata). The compaction trigger then sat several times above the
window the server actually serves and never fired — reproducing the
original #57275 'blows past the limit' symptom on current main.
Live repro (real imports, temp HERMES_HOME, config = {model:
{ollama_num_ctx: 65536}}, probed window 262144):
before: _ollama_num_ctx=65536, compressor.context_length=262144,
threshold_tokens=196608 (300% of the served window)
after: compressor.context_length=65536, threshold below the window
The clamp is one-directional (a num_ctx larger than the resolved window
never inflates the compressor) and reuses update_model() so every
threshold-derived budget recalibrates. Overlaps #60103 (silent-clamp
dead zone) — this is the init-order half.
Reported by @Artemonim in #57275 (residual claim 3).
Real windows-latest coverage for the #88136 salvage: probes drive the
actual _git_is_trampoline/_locate_real_git/_ensure_non_trampoline_git
helpers against the runner's genuine Git-for-Windows install plus a real
fork-bomb-guard trampoline stand-in. Wired into the on-demand
windows-venv-e2e lane (wine2e/** pushes only).
Review feedback on #88136 (monerostar): a profile-scoped `hermes update`
sets HERMES_HOME to <root>/profiles/<name>, but the Hermes-managed
PortableGit tree lives under the SHARED root (<root>/git/...). The locator
checked get_hermes_home() only, so a broken trampoline during a
profile-scoped update was not swapped and fell through to ZIP.
Extract _portable_git_candidates() (shared root first, profile home as
fallback) and add a regression test for the profile layout.
A Git-for-Windows trampoline launcher (bin\git.exe / cmd\git.exe shim,
~46KB) that fails to re-exec the real git-core binary refuses every git
call with a "BUG (fork bomb)" guard instead of running it (#87876).
Detect the trampoline up front via `git --version`, locate a real git
binary (Git for Windows or Hermes-managed PortableGit locations), and
rebuild the git command with it so fetch/pull/checkout keep working with
a real git instead of degrading to the ZIP fallback. When no real binary
can be found, leave the command untouched so the existing fetch-failure
handler still falls back to the ZIP path on Windows (#88046).
Google Gemini/Gemma overflow errors read 'Unable to submit request because
the input token count is 32825 but model only supports up to 32768'.
parse_context_limit_from_error had no pattern for the 'supports up to N'
phrasing, so overflow recovery kept the wrong window and burned its retry
attempts instead of recalibrating to the provider-reported limit.
Add the anchored pattern (limit follows 'supports up to'; the larger input
count before it is never captured) plus regression tests covering the exact
message and the get_context_length_from_provider_error recalibration path.
Reported by @Artemonim in #57275 (residual claim 5).
The MINIMUM_CONTEXT_LENGTH floor in _compute_threshold_tokens only
degraded to the 85% trigger when it met or exceeded the effective
window exactly (#14690). Near-minimum windows slipped through: at
context_length=65536 the threshold passed through at 64,000 — 97.7%
of the window, ~1.5K tokens of output room — so pre-API compaction
effectively could not fire.
Providers that silently truncate over-window prompts instead of
rejecting them (e.g. ollama's OpenAI-compatible /v1 endpoint) never
deliver the reactive context-overflow backstop either. Observed live
on a 65,536-token local model: the session rode into the window
ceiling and each length-continuation retry re-sent a window-filling
prompt (65,120 -> 65,273 prompt tokens, 263 output tokens of room)
until the turn died with "Response remained truncated after 4
continuation attempts" — every retry paying a full multi-minute
prefill.
Cap the floored threshold at _MIN_CTX_TRIGGER_RATIO (85%) of the
effective input budget whenever the floor is the binding term. An
explicit threshold_percent above 85% is user intent and stays
uncapped; windows where the floor lands at/below the cap are
unchanged.
Builds on webtecnica's escape-aware _split_key_path (#84152, cherry-picked
with authorship preserved; earliest fix in the family was RelaxJonh's #80253
greedy-match approach — both behaviors now ship together):
- _greedy_literal_match: when navigating an EXISTING mapping, prefer an
existing literal key equal to the dot-join of the next N path segments
(longest match wins). Dotted model IDs are the norm, so the common
unescaped command (config set providers.p.models.grok-4.6.supports_vision
true) now hits the real key across set/get/unset instead of creating a
phantom sibling. Plain dotted paths with no dotted-key collision split
exactly as before.
- _phantom_sibling + ValueError in _set_nested: refuse to CREATE a new
intermediate mapping that would shadow an existing dotted literal sibling
(Soju06's fail-loudly suggestion on #84064); set_config_value surfaces it
as a clean CLI error with the escaped spelling to use.
- utils.py::atomic_roundtrip_yaml_update (the second split site, #91607 —
/model + TUI persistence) now uses the same escape-aware split + greedy
literal matching.
- CFG-04 empty-segment guard now splits escape-aware so escaped keys are
not misclassified.
- Tests for every repro shape in the family: #84064 provider model keys,
#80006 Matrix room IDs, #91095 dotted models under custom_providers list
index (incl. escaped creation-when-absent), #91607 model_overrides via
atomic_roundtrip_yaml_update, #99124 dotted leaf keys; plus
backward-compat coverage. Also fixed the carrier's one stale assertion
(structured-value coercion landed on main after #84152 branched) and
removed its dead _MCP_SECRETS_CONFIG fixture flagged in review.
- Docs: 'Dots inside key names' section in website/docs/reference/cli-commands.md.
Fixes#84064, fixes#80006, fixes#91095, fixes#91607, fixes#99124
A cached catalog was held for the life of the process, so a long-lived
gateway or desktop kept offering models the org had since blocked until
restart. Opt-in TTL — other providers keep no-expiry caching.
Follow-up to the salvaged #98571: forward the interrupt to the compute
host whenever the parent 'running' mirror is stale, but only for
sessions that actually have hosted activity — HostSupervisor.interrupt()
calls start(), so an unconditional forward would spawn a compute-host
child just to deliver an interrupt for an idle lazy session.
Adds a regression test asserting the idle-lazy-session no-spawn path.
Refs #92916
caya8205-2