Commit Graph

13645 Commits

Author SHA1 Message Date
rainbowgits 71256dfd01 fix(state): fail loudly when state.db is replaced under a live process
Detect same-inode cp via a generation stamp, halt FTS repair, and divert
unwritten transcripts to sessions/<id>.jsonl plus the gateway pending spool.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-31 14:02:55 -07:00
Shannon Sands 5a264f9a58 docs+test: pin lease worst-case trade-off and cover all arm sites (OOF-298 review follow-ups)
Addresses Enough1122's two non-blocking review notes on PR #92316:

1. Lease vs deadline arithmetic: the state.db init/migration/repair leases
   (600-900s) are authoritative against the 300s default deadline by design.
   Add explicit comments at all three lease sites pinning the trade-off:
   single lease is deliberate (clamped to _MAX_LEASE_S=900), honest worst
   case is up to the lease duration of zombie time on a wedged DB phase,
   accepted over per-chunk renewal complexity in the migration loops.

2. Arm-site coverage: add a structural contract test asserting every
   documented entry point (hermes_cli/main.py argv fast-path,
   hermes_cli/gateway.py config-bridge re-arm, gateway/run.py backstop,
   cli.py legacy --gateway) actually calls arm_startup_watchdog (or its
   aliased import), so a future entry point can't silently ship unwatched.
2026-08-31 14:01:39 -07:00
Shannon Sands 852db61abe fix(startup-watchdog): bounded hard-exit escort + phase-owned progress leases
Addresses the two class-level review blockers on PR #89750:

1. Bounded hard-exit seam (escort thread). The forensic fire path
   (logger.critical, dump record, faulthandler, lifecycle ledger) can
   itself wedge — the parked main thread may hold the logging handler
   lock, or the disk may be full/hung. _fire() now starts an exit-escort
   daemon thread BEFORE any forensics; it is free of log handlers,
   filesystem access, module loads and application locks, and hard-exits
   with the restart code after _FIRE_EXIT_BOUND_S unless the normal fire
   path signals completion. Adversarial tests hold the logging handler
   lock / hang the dump write at fire time and assert the exit seam is
   still reached.

2. Phase-owned progress leases (report_startup_progress). Process CPU
   time proves process activity, not startup progress: an unrelated busy
   thread could extend forever while startup sits parked (false
   negative), and I/O-bound repair/backup accrues ~zero CPU and would be
   killed (false positive). Long synchronous startup phases now declare
   authoritative, clamped (_MAX_LEASE_S), renewable progress leases:
   state.db _init_schema + the version-gated data-migration chain
   (hermes_state_schema) and repair_state_db_schema (hermes_state) are
   wired. CPU progress remains only as a bounded fallback, capped at
   _MAX_CPU_EXTENSIONS, with leases outranking the cap. Adversarial
   tests cover both directions (lease saves zero-CPU legitimate work;
   capped CPU noise no longer hides a parked deadlock).

Fire-path dump record now includes lease_count/last_lease_phase for
forensics. gateway/startup_watchdog.py shim re-exports
report_startup_progress.

OOF-298
2026-08-31 14:01:39 -07:00
Shannon Sands f5bb1e144d fix(gateway): address startup-watchdog review findings (OOF-298, PR #89750)
Independent review of the initial startup-liveness watchdog surfaced two
P1s and three P2s. All are addressed here.

P1 — legitimate slow startups (large state.db schema migrations inside
SessionDB.__init__, which run synchronously before the loop starts) could
exceed the fixed 300s deadline and restart-loop. The watchdog now checks
process CPU time (time.process_time(), process-wide) when the deadline
expires: continuous CPU consumption means a live migration, so the deadline
is extended (with a warning log per extension). The OOF-298 deadlock class
parks every thread in futex waits and accrues ~zero CPU, so it still fires
on schedule. Documented limitation: a spinning busy-wait deadlock reads as
progress and won't fire — the observed incident class is parked threads.

P1 — import-time deadlocks were outside coverage. The implementation moved
to a stdlib-only top-level module (hermes_startup_watchdog), and
hermes_cli/main.py arms it via an argv fast-path ("gateway" + "run" in
argv) BEFORE the heavy module-level import graph. gateway/startup_watchdog
remains as a re-export shim so the intuitive import path keeps working for
the disarm site, tests, and REPL use. Import-lightness is a correctness
property, tested via AST inspection: at fire time the wedged main thread
may hold the import lock, so the fire path performs no imports on its own
thread — the lifecycle-ledger write runs on a bounded-join helper thread
and os._exit happens regardless.

P2 — disarm/fire race: the handle now has an explicit state machine
(armed → disarmed | firing) guarded by a lock; whichever transition takes
the lock first wins, so a disarm landing after deadline expiry but before
the fire transition is honored. Regression test forces the exact
interleaving by blocking inside the CPU probe.

P2 — uncovered entry points: cli.py --gateway and scripts/hermes-gateway
run_gateway() now arm the watchdog before importing the gateway graph.
hermes_cli/gateway.py run_gateway() keeps an idempotent backstop arm for
programmatic callers.

P2 — respawn-storm backoff interaction: the storm breaker's intentional
backoff sleep (up to minutes, ~zero CPU — indistinguishable from a parked
deadlock) now calls kick_startup_watchdog(extra_s=backoff) so the deadline
is pushed past the sleep instead of firing mid-backoff.

Also: the faulthandler stack dump is now additionally written to
logs/gateway-startup-watchdog.log (stderr may be absent on detached/
windowless runs); the disarm site in gateway/run.py moved inside the
loop-confirmed branch (if the loop is NOT live, the milestone was not
reached and the watchdog must stay armed); hermes_startup_watchdog added
to pyproject py-modules so sealed venvs ship it; SERVICE_RESTART_EXIT_CODE
is duplicated in the stdlib-only module with a parity test against
gateway.restart.

Tests: 38 in tests/gateway/test_startup_watchdog.py (contracts incl.
stdlib-only AST check and shim re-export identity, config resolution,
arm/disarm/kick, CPU-progress extension vs no-progress fire, probe-failure
fails toward firing, disarm-vs-fire race, dump record + file stacks,
lifecycle ledger, custom exit code).
2026-08-31 14:01:39 -07:00
Shannon Sands 8a3b6f374d fix(gateway): startup-liveness watchdog for pre-event-loop deadlocks (OOF-298)
A hosted gateway (hermes-doubleam-2568) deadlocked at startup with every
thread parked in futex_wait_queue before the asyncio loop came alive:
zero log lines, /health unreachable — but s6 saw a live PID so it never
respawned the process, and a stale gateway_state.json from the previous
life told every status surface "draining" for ~30 hours.

Every existing liveness backstop assumes startup succeeded: the
loop-liveness watchdog is armed inside the running loop's startup path,
the shutdown watchdog arms at stop(), and the heartbeat file is written
by an asyncio task. None can fire when the process wedges before the
loop exists.

New gateway/startup_watchdog.py: a plain daemon OS thread armed at
process entry (both gateway.run.main() and the `hermes gateway run` CLI
wrapper), disarmed the moment GatewayRunner confirms a live event loop —
the point where the existing loop-liveness watchdog takes over. If
startup neither reaches that milestone nor exits within the deadline
(default 300s; slowest legitimate pre-loop work is the 120s-bounded MCP
discovery wait), the watchdog:

* dumps all-thread stacks via faulthandler,
* appends a JSON record to logs/gateway-startup-watchdog.log,
* records the exit in the NS-608 lifecycle ledger
  (reason=startup_liveness_watchdog) so the next boot classifies it
  instead of reporting an unclean SIGKILL/OOM death,
* os._exit(75) so s6/systemd respawn the process.

Config is env-only (HERMES_STARTUP_WATCHDOG=0 to disable,
HERMES_STARTUP_WATCHDOG_TIMEOUT_S to tune, floor-clamped to 30s):
the watchdog must be armed before config.yaml is loaded — a wedge
during config parsing is exactly in scope — so it cannot depend on
config for its own enablement. Everything is best-effort; a watchdog
failure never affects the startup it observes.

Arm sites are placed after the PID-file/--replace conflict guards so a
--replace loser exiting early never arms a watchdog. Disarm happens even
when the loop guards are config-disabled (gateway.loop_watchdog: false)
— the startup watchdog only covers the pre-loop window, never adapter
connects or steady-state, so WhatsApp pairing / npm cold installs are
unaffected.

Tests: tests/gateway/test_startup_watchdog.py (29 tests — config
resolution, arm/disarm idempotency, fire path with captured exit,
lifecycle-ledger marking, dump record, disable knob).

Fixes OOF-298.
2026-08-31 14:01:39 -07:00
Teknium 3783fd9ffe fix(update): defer ledger-verified serve/dashboard holders to the updater instead of dead-ending the Desktop hand-off
On Windows, the Desktop update preflight scan (hermes_cli._scan_venv_blockers)
classified only python -m http.server as a stoppable blocker. A hermes serve
or dashboard backend that survived the Desktop teardown (or was launched
manually) either dead-ended the hand-off with venv-blocked, or slipped past
it and kept venv\Scripts\hermes.exe mapped, so the updater quarantine
failed with os error 32 (#98336).

The CLI updater downstream already owns exactly this holder class with
positive-identity rungs: _ledger_reapable_backend_pids reaps dead-spawner
orphans and _ledger_manual_serve_holders stops manual serves and relaunches
them on their recorded host/port. Mirror the existing pausable-gateway
exemption: defer ledger-verified serve/dashboard holders to those rungs
instead of reporting them as blockers.

Identity is positive-only, per the #99558 guard contract: token-parsed
subcommand (never substring, #90778) + live-verified (pid, create_time)
ledger entry with matching purpose + provable ownership (spawner dead,
unrecorded, or the hand-off Desktop itself — an ancestor of the scan,
verified by pid+create_time). A backend supervised by any other live
process keeps blocking; ledger unreadable fails closed.

Fixes #98336
2026-08-31 13:11:54 -07:00
Teknium fb9b2c893f feat(agent): escalate repeated transcript-sanitiser heals with a one-time user notice (#96870)
Builds the escalation layer on top of HexLab98's heal-log windowing
(salvaged from PR #96916):

- Per-session heal counters (heal events + messages healed) tracked by the
  repair path in agent_runtime_helpers.py, session totals preserved across
  10-minute log windows.
- Threshold escalation: after N heals in a session window (default 3,
  configurable via agent.sanitizer_heal_escalation_threshold in
  config.yaml, 0 = off) log ONE ERROR carrying session id + heal pattern
  (events/messages/window/threshold), then stay quiet.
- ONE-TIME out-of-band user notice queued at the threshold and delivered by
  the conversation loop through _emit_warning (status callback -> gateway
  status message / CLI print). Never injected into conversation context or
  the wire copy: prompt caching, role alternation, and durable history are
  untouched. Never re-arms on a new window; scoped per session.
- Counters visible in diagnostics: get_sanitizer_heal_stats() rendered in
  the /debug share // hermes debug report, and the config key surfaced in
  hermes dump overrides. errors.log carries the ERROR line for `hermes logs
  errors`.
2026-08-31 13:11:41 -07:00
HexLab98 a779f527fa test(agent): cover empty-transcript projection fill and heal-log escalation (#96870) 2026-08-31 13:11:41 -07:00
theo 8207862212 fix(compression): stop timeout paths from blocking retries 2026-08-31 13:00:33 -07:00
theo 0a8b25e0ab fix(auxiliary): forward service tier on Codex Responses 2026-08-31 13:00:33 -07:00
Teknium 13c3958df5 fix(auxiliary): credit-limited 402s clamp to the affordable budget instead of failing
Second leg of the masoria 20-minute 'Summarizing' stall (Aug 31 2026
bundle): after the Codex timeout, compression fell back to OpenRouter,
which defaulted the omitted output cap to the model's full 65,536-token
window and rejected with '402 ... can only afford 7117' — on an account
whose balance easily covered a summary. Three fallbacks, three 402s,
zero summaries.

- _create_with_progress: when a 402 names an affordable budget, retry
  ONCE with that cap (minus 64-token margin, 512-token floor); plain
  exhaustion 402s and within-budget 402s re-raise unchanged. Single
  funnel covers primary, fallback, and retry call sites.
- extends the #41055 OpenRouter max_tokens preservation onto the current
  _build_call_kwargs gate (explicit caps survive; None still omitted).

Live A/B against a mock OpenRouter enforcing a 7117-token budget with
the real SDK + real adapter: main fails with the exact bundle 402;
fixed branch retries clamped and returns the summary.
2026-08-31 13:00:24 -07:00
liuhao1024 b26a1eae8f fix(auxiliary): preserve max_tokens for OpenRouter to prevent free-tier 402
_build_call_kwargs strips max_tokens for non-Anthropic providers to
avoid wire-format issues (Copilot, ZAI, GPT-5). However, OpenRouter
free/limited-credit tiers need max_tokens because the model's full
output window exceeds the credit budget, causing HTTP 402.

Without max_tokens, the 402 triggers fallback to a text-only model
which then fails with 'unknown variant image_url, expected text'.

Include max_tokens when provider is 'openrouter' or base_url contains
'openrouter.ai'.

Fixes #41035
2026-08-31 13:00:24 -07:00
Teknium 093d1ebd0b test(auxiliary): watchdog timeout asserts owner-thread FD release
Adapt the #99660 blocked-before-first-event watchdog test to the FD-ownership
contract: the stranger-thread Timer marks the timeout but never close()s;
the owning thread surfaces the TimeoutError and releases the FDs on unwind.
2026-08-31 13:00:18 -07:00
dsad 8a5b49d86a fix(auxiliary): never release Codex client FDs from the timeout Timer
_CodexCompletionsAdapter.create arms a daemon threading.Timer that calls
client.close() when the aux Responses stream exceeds its timeout. On a
stalled stream -- the failure the timeout exists for -- the Timer is the
only thing that fires, so the close runs on a thread that does not own
the in-flight httpx connection.

That is the FD-ownership violation the repo already fixed twice on the
main transport (#29507, #67142, #70773): close() releases the raw TLS fd
while the owner's OpenSSL BIO still caches that integer, the kernel
recycles it into the next open() in the process -- a SessionDB or
kanban.db handle -- and the owner's unwinding TLS flush writes an
application-data record into that database file.

agent/auxiliary_client.py had no thread-ownership machinery at all: the
guarded twins (_retire_shared_openai_client, _abort_request_openai_client)
live in run_agent.py and are unreachable from this adapter, which holds no
AIAgent reference.

Dispatch on ownership the way chat_completion_helpers already does: from a
stranger thread only force_close_tcp_sockets() (shutdown(SHUT_RDWR), which
is FD-safe from any thread), and let the owning thread release the FDs when
it unwinds. The owner-thread caller (_check_cancelled) keeps closing
directly. Cache eviction (#23432) is unchanged.
2026-08-31 13:00:18 -07:00
fangliquanflq 9caf31394f fix(update): verify repaired current-checkout runtime 2026-08-31 12:58:14 -07:00
fangliquanflq 4f19808915 fix(update): fail current-checkout runtime verification 2026-08-31 12:58:14 -07:00
fangliquanflq 7e22725bb4 fix(update): verify SQLite runtime remediation 2026-08-31 12:58:14 -07:00
Finn763 57f653db90 test(managed_uv): make self-lock regression assert carry its message
The previous form was `mock_install.assert_not_called(), ("msg")` — a bare
tuple expression whose parenthetical never surfaces as a failure message.
Switch to `assert mock_install.call_count == 0, "msg"` so the diagnostic
actually appears when the guard regresses (review feedback on #93163).
2026-08-31 12:50:29 -07:00
Finn763 d0503b500f fix(update): defer Windows runtime repair when the updater holds the venv
On Windows, hermes update launched from the install's own venv can never
complete the managed-runtime repair: Windows keeps the image of the
updater's venv\Scripts\python.exe (and the waiting hermes.exe launcher
ancestor) mapped until exit, so the park rename in _cut_over_candidate
always fails with ERROR_ACCESS_DENIED. The pre-flight holder scan
deliberately excludes the calling process and its ancestors, so the guard
passes and the repair burns its retries on a structurally unwinnable
rename - forever, since the failure is non-fatal and the 'next update
will retry' message is misleading for this case.

Detect the self-lock (sys.executable or a launcher ancestor inside the
live venv) before provisioning and defer with actionable guidance instead
of walking into the doomed cutover. The deferral happens pre-provisioning,
so the incomplete generation-* leftovers the reporter observed are no
longer produced. No-op off Windows: POSIX renames work while the updater
maps the tree. Mirrors the existing _defer_update_for_self_lock pattern.

Regression tests prove the fix bites: neutralized guard -> repair proceeds
to provisioning (red); restored guard -> deferred before provisioning
(green). Verified on Windows 11 against a real venv.

Closes #93032
2026-08-31 12:50:29 -07:00
Kyzcreig c1bd0511cf fix(gateway): persist the platform message id on every user turn 2026-08-31 12:46:32 -07:00
Teknium 5505042f40 fix(sessions): fail closed on ownership uncertainty and fence every turn source
Follow-ups on top of the #94595 cherry-pick, implementing the maintainer
review's two blockers:

Blocker 1 (turn-admission chokepoint): _run_prompt_submit itself now runs
the ownership admission, so synthesized turns that never pass through the
prompt.submit RPC handler (crash auto-continue from cold session.resume,
wake-ups) are fenced too. Auto-continue additionally checks ownership
BEFORE emitting message.start and leaves the marker in place, closing the
#94778 shape where backend B resumed a session backend A was actively
running and auto-continued A's fresh interrupted-turn marker into a
duplicate concurrent turn.

Blocker 2 (fail-closed registry semantics): try_acquire_active_session no
longer converts an unreadable/corrupt registry into an untracked go-ahead.
Ownership uncertainty is a distinct typed refusal —
SESSION_COORDINATION_UNAVAILABLE — because "could not prove ownership" must
never be collapsed into "no owner exists". The TUI gateway claim helper
fails closed on claim exceptions for every surface, not just desktop.

Also: empty session ids short-circuit to a no-op lease (nothing to fence,
and the strict registry schema rejects empty ids), and the existing
fail-open tests were updated to assert the new fail-closed contract.
2026-08-31 12:36:33 -07:00
Futahua a5f0fbb262 fix(sessions): per-session exclusivity is correctness, not a capacity policy
Cherry-picked from PR #94595 (author: Futahua) onto current main, with the
maintainer-review revision points folded in during the rebase:

- the lease engages UNCONDITIONALLY: try_acquire_active_session no longer
  returns a disabled no-op lease when max_concurrent_sessions is unset;
  the concurrency cap stays an orthogonal, optional policy checked second
- ownership uncertainty fails CLOSED (SESSION_COORDINATION_UNAVAILABLE)
  instead of degrading to an untracked go-ahead: a corrupt/unreadable
  registry must not be collapsed into 'no owner exists' (review blocker 2)
- the ownership admission sits at the _run_prompt_submit chokepoint that
  EVERY fresh turn source crosses, and crash auto-continue acquires (or
  bails) BEFORE emitting message.start — closing the #94778 bypass where
  backend B's auto-continue ran a duplicate turn while backend A was live
  (review blocker 1)
- the TUI gateway claim helper fails closed on claim exceptions for every
  surface, not just desktop
- CLI and messaging-gateway call sites pass live_session_id metadata so
  the (pid, live id) re-entrancy identity protects them from self-fencing
  on a leaked lease

Co-authored-by: teknium1 <teknium1@users.noreply.github.com>
2026-08-31 12:36:33 -07:00
fangliquanflq 53c0df6de9 fix(agent): stop compression retries after host timeout (#98722)
Salvaged from #98741, composed on top of the merged #98424 preflight
fail-closed boundary. A host-ceiling compression timeout is now a typed,
thread-safe outcome consumed by every automatic caller:

- conversation_compression.py: threading.local + per-agent lock timeout
  state (mark/reset/read helpers) upgrading #98424's simple attribute
  where overlapping automatic/manual compression entrypoints matter;
  the _last_compression_timed_out attribute stays as compat mirror.
- conversation_loop.py: the mid-turn pre-API pass and the provider
  overflow (413/400 context_length_exceeded) recovery path end the turn
  with the typed compression_exhausted recovery contract instead of
  re-sending the unchanged oversized request and re-entering compression
  in the same turn.
- run_agent.py/turn_context.py: forwarder resets the typed state per
  attempt; the #98424 turn-start check reads it through the typed helper.

Tests: thread-safety/atomicity of the state helpers, overflow-recovery
non-re-entry, and typed terminal result.
2026-08-31 12:36:02 -07:00
Dev 532ee1a199 fix(gateway): persist gateway routing identity on lazily-created session rows
When the default/global state.db is corrupt at gateway startup,
SessionStore degrades (_db=None) and record_gateway_session_peer never
self-heals. Under multiplexed profile routes the AIAgent's lazy
_ensure_db_session was then the ONLY durable write for the session, and
it created the row identity-less (session_key/chat_id/chat_type/
thread_id/user_id/origin_json all NULL) — unrecoverable by
find_latest_gateway_session_for_peer, so Telegram chats forgot prior
turns.

Persist the routing identity the agent already carries into
create_session; plain CLI sessions keep the old identity-less shape.

Salvaged leg 1 of #88804; the transcript-recovery leg is covered by the
scope-aware session DB resolution already on main.
2026-08-31 12:32:33 -07:00
Teknium 7790c8f4c4 fix(telegram): bound stale-client cleanup and add Windows CLOSE-WAIT live probes (#87057)
Follow-ups on top of the salvaged commits from PR #87111 (@HexLab98) and
PR #87265 (@JoaoMarcos44):

- keep main's #92991 stall watchdog (150s progress-based) as the single
  steady-state liveness probe instead of adding a second overlapping one
- orphaned-client aclose() cleanup uses the wall-clock thread deadline and
  is tracked in _background_tasks so a wedged close can neither hang nor
  leak one task per reconnect attempt (from #87265's review findings)
- merge #87265's no-keepalive getUpdates pool (max_keepalive_connections=0)
  with #87111's TCP-keepalive socket options on all transports
- add tests/gateway/test_telegram_closewait_windows_live.py: live probes
  against a real half-closing HTTP server, skipif non-win32, wired into
  the on-demand windows-venv-e2e lane (wine2e/**)
2026-08-31 12:28:49 -07:00
joaomarcos b02b7122f8 fix(telegram): prevent Windows long-poll socket reuse deadlock
Prevent the dedicated getUpdates pool from reusing server-closed connections and replace a polling HTTP client left open after a timed-out CLOSE-WAIT drain. Keep the general Bot API pool reusable so concurrent sends and edits are unaffected. Add regression coverage for both transport limits and stale-client replacement. Fixes #87057
2026-08-31 12:28:49 -07:00
HexLab98 505f64b798 test(telegram): cover CLOSE-WAIT drain rebuild and getUpdates liveness 2026-08-31 12:28:49 -07:00
Teknium 7cefa87ea7 fix(agent_init): reserve Gemini's default maxOutputTokens in the compressor when max_tokens is unset
The native generateContent adapter never runs uncapped: when
model.max_tokens is unset it sends maxOutputTokens=65,535
(GEMINI_DEFAULT_MAX_OUTPUT_TOKENS) because Gemini treats an omitted cap
as a low internal default. The context compressor's trigger is
pct×(window − max_tokens), and constructing it with max_tokens=None
reserved 0 — so on a 128K Gemma window the trigger landed at 98,304
while the real safe input budget was 65,537, and the provider 400'd
before compaction fired.

Live repro (real imports, temp HERMES_HOME, native Gemini base_url,
window=131072, max_tokens unset):
  before: compressor.max_tokens=None, threshold_tokens=98304,
          wire maxOutputTokens=65535 → trigger ABOVE the safe budget
  after:  compressor.max_tokens=65535, threshold_tokens=64000 → below it

Scoped to the native Gemini wiring (provider names + native base_url via
is_native_gemini_base_url; the /openai compat endpoint is excluded). The
generic provider-default reservation gap remains tracked in #63839.

Reported by @Artemonim in #57275 (residual claim 4).
2026-08-31 12:22:55 -07:00
Teknium cb71d5f1b1 fix(agent_init): clamp compressor window to Ollama num_ctx resolved after construction
model.ollama_num_ctx is resolved AFTER the context compressor is
constructed, so a config that sets only ollama_num_ctx (without
model.context_length) ran every request at the smaller served num_ctx
while the compressor still targeted the probed GGUF window (e.g. 256K
Gemma metadata). The compaction trigger then sat several times above the
window the server actually serves and never fired — reproducing the
original #57275 'blows past the limit' symptom on current main.

Live repro (real imports, temp HERMES_HOME, config = {model:
{ollama_num_ctx: 65536}}, probed window 262144):
  before: _ollama_num_ctx=65536, compressor.context_length=262144,
          threshold_tokens=196608 (300% of the served window)
  after:  compressor.context_length=65536, threshold below the window

The clamp is one-directional (a num_ctx larger than the resolved window
never inflates the compressor) and reuses update_model() so every
threshold-derived budget recalibrates. Overlaps #60103 (silent-clamp
dead zone) — this is the init-order half.

Reported by @Artemonim in #57275 (residual claim 3).
2026-08-31 12:22:17 -07:00
Teknium 7acc65a399 test(windows): live E2E for the git trampoline self-heal on the wine2e lane
Real windows-latest coverage for the #88136 salvage: probes drive the
actual _git_is_trampoline/_locate_real_git/_ensure_non_trampoline_git
helpers against the runner's genuine Git-for-Windows install plus a real
fork-bomb-guard trampoline stand-in. Wired into the on-demand
windows-venv-e2e lane (wine2e/** pushes only).
2026-08-31 12:21:46 -07:00
SayHell0W0rld 6cccc2ef7e fix(update): locate PortableGit under the shared root, not profile home
Review feedback on #88136 (monerostar): a profile-scoped `hermes update`
sets HERMES_HOME to <root>/profiles/<name>, but the Hermes-managed
PortableGit tree lives under the SHARED root (<root>/git/...). The locator
checked get_hermes_home() only, so a broken trampoline during a
profile-scoped update was not swapped and fell through to ZIP.

Extract _portable_git_candidates() (shared root first, profile home as
fallback) and add a regression test for the profile layout.
2026-08-31 12:21:46 -07:00
SayHell0W0rld 0bab9ff9e7 fix(update): self-heal broken Git-for-Windows trampoline on Windows
A Git-for-Windows trampoline launcher (bin\git.exe / cmd\git.exe shim,
~46KB) that fails to re-exec the real git-core binary refuses every git
call with a "BUG (fork bomb)" guard instead of running it (#87876).

Detect the trampoline up front via `git --version`, locate a real git
binary (Git for Windows or Hermes-managed PortableGit locations), and
rebuild the git command with it so fetch/pull/checkout keep working with
a real git instead of degrading to the ZIP fallback. When no real binary
can be found, leave the command untouched so the existing fetch-failure
handler still falls back to the ZIP path on Windows (#88046).
2026-08-31 12:21:46 -07:00
LucidPaths 0943702c55 fix(gateway): keep long turns controllable without blocking Telegram 2026-08-31 12:21:01 -07:00
Teknium 58f5b1e277 fix(model_metadata): parse Google's 'supports up to N' context-limit phrasing
Google Gemini/Gemma overflow errors read 'Unable to submit request because
the input token count is 32825 but model only supports up to 32768'.
parse_context_limit_from_error had no pattern for the 'supports up to N'
phrasing, so overflow recovery kept the wrong window and burned its retry
attempts instead of recalibrating to the provider-reported limit.

Add the anchored pattern (limit follows 'supports up to'; the larger input
count before it is never captured) plus regression tests covering the exact
message and the get_context_length_from_provider_error recalibration path.

Reported by @Artemonim in #57275 (residual claim 5).
2026-08-31 12:20:02 -07:00
Darafei Praliaskouski 4252aecc2e fix(agent): cap compaction threshold floor at 85% of the context window
The MINIMUM_CONTEXT_LENGTH floor in _compute_threshold_tokens only
degraded to the 85% trigger when it met or exceeded the effective
window exactly (#14690). Near-minimum windows slipped through: at
context_length=65536 the threshold passed through at 64,000 — 97.7%
of the window, ~1.5K tokens of output room — so pre-API compaction
effectively could not fire.

Providers that silently truncate over-window prompts instead of
rejecting them (e.g. ollama's OpenAI-compatible /v1 endpoint) never
deliver the reactive context-overflow backstop either. Observed live
on a 65,536-token local model: the session rode into the window
ceiling and each length-continuation retry re-sent a window-filling
prompt (65,120 -> 65,273 prompt tokens, 263 output tokens of room)
until the turn died with "Response remained truncated after 4
continuation attempts" — every retry paying a full multi-minute
prefill.

Cap the floored threshold at _MIN_CTX_TRIGGER_RATIO (85%) of the
effective input budget whenever the floor is the binding term. An
explicit threshold_percent above 85% is user intent and stays
uncapped; windows where the floor lands at/below the cap are
unchanged.
2026-08-31 12:19:29 -07:00
Teknium a42aee9585 fix(config): greedy literal-key matching + loud phantom-sibling refusal for dotted key names
Builds on webtecnica's escape-aware _split_key_path (#84152, cherry-picked
with authorship preserved; earliest fix in the family was RelaxJonh's #80253
greedy-match approach — both behaviors now ship together):

- _greedy_literal_match: when navigating an EXISTING mapping, prefer an
  existing literal key equal to the dot-join of the next N path segments
  (longest match wins). Dotted model IDs are the norm, so the common
  unescaped command (config set providers.p.models.grok-4.6.supports_vision
  true) now hits the real key across set/get/unset instead of creating a
  phantom sibling. Plain dotted paths with no dotted-key collision split
  exactly as before.
- _phantom_sibling + ValueError in _set_nested: refuse to CREATE a new
  intermediate mapping that would shadow an existing dotted literal sibling
  (Soju06's fail-loudly suggestion on #84064); set_config_value surfaces it
  as a clean CLI error with the escaped spelling to use.
- utils.py::atomic_roundtrip_yaml_update (the second split site, #91607 —
  /model + TUI persistence) now uses the same escape-aware split + greedy
  literal matching.
- CFG-04 empty-segment guard now splits escape-aware so escaped keys are
  not misclassified.
- Tests for every repro shape in the family: #84064 provider model keys,
  #80006 Matrix room IDs, #91095 dotted models under custom_providers list
  index (incl. escaped creation-when-absent), #91607 model_overrides via
  atomic_roundtrip_yaml_update, #99124 dotted leaf keys; plus
  backward-compat coverage. Also fixed the carrier's one stale assertion
  (structured-value coercion landed on main after #84152 branched) and
  removed its dead _MCP_SECRETS_CONFIG fixture flagged in review.
- Docs: 'Dots inside key names' section in website/docs/reference/cli-commands.md.

Fixes #84064, fixes #80006, fixes #91095, fixes #91607, fixes #99124
2026-08-31 12:18:47 -07:00
webtecnica 8951947531 fix(cli): support literal dots in config set/unset key paths (#84064) 2026-08-31 12:18:47 -07:00
Teknium 4396253aed fix(gateway): gate compute-host interrupt forward on hosted activity
Follow-up to the salvaged #98571: forward the interrupt to the compute
host whenever the parent 'running' mirror is stale, but only for
sessions that actually have hosted activity — HostSupervisor.interrupt()
calls start(), so an unconditional forward would spawn a compute-host
child just to deliver an interrupt for an idle lazy session.

Adds a regression test asserting the idle-lazy-session no-spawn path.

Refs #92916
2026-08-31 12:18:00 -07:00
konsisumer 3ae74119c9 fix(gateway): relay compute-host clarify state 2026-08-31 12:18:00 -07:00
the3asic 18ac3c4fb6 fix(state): defer corrupt FTS rebuilds past live operations 2026-08-31 12:08:30 -07:00
devops ad08a5813c fix(estop): honor canonical ~/.hermes/ESTOP from profile gateways
Profile processes launch with HERMES_HOME=~/.hermes/profiles/<name>, so
`hermes pause` at the fleet root did not bind fleet-analyst dispatch
(t_7b65ff88). Check/resume both the process home and the fleet root.
2026-09-01 00:35:29 +05:30
Teknium b0acc558b1 fix(cli): supervised gateway launches skip the sticky active_profile redirect
Generalize the HERMES_S6_SUPERVISED_CHILD supervisor-marker mechanism so
ANY supervised gateway launch (systemd, launchd, Windows Scheduled Task,
external supervisor) skips the active_profile redirect in
_apply_profile_override(). Previously only the s6 container marker was
honored, so a systemd-launched default-profile gateway with
HERMES_HOME=<root> followed the sticky active_profile file and silently
assumed another profile's identity — logging under that profile's tree
and connecting with its Telegram bot token (double-polling a token owned
by that profile's own live gateway).

- hermes_cli/main.py: honor HERMES_SUPERVISED_CHILD (new generalized
  marker), HERMES_S6_SUPERVISED_CHILD (back-compat), INVOCATION_ID
  (systemd; gateway commands only, since it leaks into every descendant
  of systemd-launched processes), and HERMES_GATEWAY_EXTERNAL_SUPERVISOR.
- hermes_cli/gateway.py: export HERMES_SUPERVISED_CHILD=1 in generated
  systemd units (user + system) and the launchd plist.
- hermes_cli/gateway_windows.py: export it from the Scheduled-Task cmd/vbs
  launchers and the windowless respawn env overlay.
- hermes_cli/service_manager.py: export it alongside the s6 sentinel.
- tests: regression coverage for all markers + non-gateway INVOCATION_ID
  neutrality + generated-unit marker presence.

Fixes #74872
2026-08-31 12:05:07 -07:00
Teknium 11ba76c0fb fix(gateway): run MCP shutdown off-loop with a bounded wait on the shutdown path
shutdown_mcp_servers() blocks on future.result(timeout=15) which, called
from the gateway event-loop thread during SIGTERM teardown, freezes the
loop for up to 15s when the MCP loop and its stdio children are torn down
concurrently. Supervisors with a shorter kill grace (s6-overlay: 3s)
SIGKILL the gateway before lifecycle_ledger.mark_exited() runs, producing
phantom 'exited UNCLEANLY' reports on every subsequent boot.

Run the sync shutdown on a daemon thread and poll via _await_thread_exit
with a 5s budget; proceed with teardown if it wedges. Fixes #82874;
completes the shutdown half of #64155.
2026-08-31 12:04:10 -07:00
Teknium 51773a7733 test(state): physical-corruption acceptance tests for the fail-closed classifier
Real byte-flip fixtures (no mocks) proving the #96038/#98090-class fix
end to end, closing the acceptance gate on issue #97940:

- test_canonical_btree_corruption_fails_closed: checkpoint the WAL,
  clobber every messages-table B-tree leaf page header, then assert a
  live append raises the genuine bare SQLITE_CORRUPT, the classifier
  refuses the FTS route, no rebuild/detach/stale-marker side effects
  occur, and the field incident's misdiagnosis log line ('canonical
  message rows are preserved') never appears.
- test_fts_only_corruption_still_self_heals: contrast case — a real
  messages_fts_data shadow-table stomp raises SQLITE_CORRUPT_VTAB (267),
  is classified as FTS-scoped, and the write path still self-heals with
  canonical rows intact.

Sabotage-verified: reverting the classifier fix (96739033c4) makes the
canonical-corruption test fail by entering the FTS self-heal route.

Credits @fangliquanflq (PR #98090) for the production timeline analysis
and @diatche (PR #96038) for the classifier fix these tests gate on.
Refs #97940, #98077.
2026-08-31 12:02:58 -07:00
andyst-dev 4d3e1e4d13 fix(desktop): prevent venv scan timeout on busy Windows hosts 2026-08-31 12:00:33 -07:00
HexLab98 85118cc9d4 test(cli): cover mixed-config ImportError recovery hint on chat startup (#96900) 2026-09-01 00:26:55 +05:30
kshitijk4poor fe0cfdf99c fix(redact): keep dotted config-key scans linear past the keyword pre-gate
The _CFG_SECRET_WORD_RE pre-gate only skips secret-FREE text. A compaction
payload containing one real secret assignment plus a long opaque dotted run
still reaches _CFG_DOTTED_RE's backtrackable '*' prefix, which re.sub retries
from every byte of the run — quadratic while holding the GIL (same class as
the _ENV_ASSIGN_LOWER_RE fix in this branch, #99255).

Anchor each attempt to the start of a key run with a negative lookbehind.
Match set is unchanged: any match starting mid-run implies a leftmost match
at the run start, verified 20/20 identical over a dotted-config corpus.
30k-char adversarial run: 102s -> 0.015s.
2026-09-01 00:25:45 +05:30
Teknium f50b5bb0fa fix(compression): dead Codex summary streams fail over in 60s instead of stacking 5-minute waits
The Codex auxiliary Responses adapter enforced a single absolute
deadline (300s floor for compression). A dead stream held the entire
budget before fallback ran, and repeated compression attempts stacked
those waits into 20+ minute 'Summarizing thread' stalls (masoria debug
bundle, Aug 31 2026). Meanwhile a healthy-but-slow reasoning summary
was killed at the same absolute deadline even while producing tokens.

Replace the absolute kill with progress-aware deadlines:
- 60s no-progress window for the first substantive payload AND between
  payloads; keepalive/lifecycle frames do not re-arm (mirrors the
  commit-fence gating, #96707)
- a live stream re-arms per token and is bounded only by
  _aux_stream_total_ceiling() (max(600s, 4x configured timeout)), the
  same backstop the streamed chat.completions path already uses
- the compression critical-path retry gate now distinguishes failure
  cost: a cheap first-token no-progress failure retries the same
  provider once; mid-stream stalls and ceiling hits still skip straight
  to provider fallback (#54465 semantics preserved)

Live A/B (real OpenAI SDK against a local SSE server, real adapter):
dead keepalive-only stream: main waits the full budget; fixed fails
over at the window. Slow-but-alive stream (tokens past the configured
timeout): main kills it mid-generation; fixed completes.
2026-08-31 11:52:51 -07:00
embwl0x 221312834e fix(gateway): bound signal interrupt grace 2026-08-31 11:47:16 -07:00
Teknium f680a4dc7d fix(gateway): require FTS provenance before transcript rebuild-and-retry
Widen #96038's fail-closed classifier to the gateway transcript retry
path: SessionStore._is_fts_corruption_error no longer treats a generic
'database disk image is malformed' as FTS-only damage. It now delegates
to SessionDB._is_fts_write_corruption_error (SQLITE_CORRUPT_VTAB result
code or explicit fts5 corrupt-structure text) and only keeps the
messages_fts-named cases. Structural corruption falls through to the
bounded retry/backoff path instead of rebuilding FTS and retrying writes
against a damaged database.

Sibling site spotted in PR #98090 by @fangliquanflq.
2026-08-31 11:42:23 -07:00