Each release exact-pins at least one dependency to a version published
days before the release (v0.20.6: snowballstemmer==3.1.1, psutil==7.2.2).
For two weeks after release the relative exclude-newer cutoff filters
those versions out, so any venv that predates the release cannot resolve
the new pins at all ('no version of snowballstemmer==3.1.1' — observed
2026-08-29 updating three production installs v0.20.0 -> v0.20.6, one
Termux and two Linux servers; the Termux host additionally bricked on
psutil==7.2.2 sdist resolution, and cryptography's isolated build
environment resolved maturin/setuptools-rust under the same cutoff).
Same zero-float-protection logic as the setuptools/pillow/mcp/
firecrawl-anydoc exemptions: the pin bump WAS the review, so the cutoff
adds nothing for an exact pin and can only brick. Extend
exclude-newer-package to every exact-pinned package in
[project].dependencies / optional-dependencies (table moved to
one-key-per-line — 97 entries), plus maturin and setuptools-rust for
wheel-less sdist builds of the exempted cryptography pin.
test_exact_pinned_deps_exempt_from_exclude_newer enforces the invariant
going forward: adding a name==version pin without a matching
exclude-newer-package entry fails CI.
Windows installer editable builds fail in uv's isolated sandbox with
ModuleNotFoundError: wheel.cli because build-system.requires only listed
setuptools. setuptools.build_meta and our setup.py bdist_wheel guard both
import wheel during the build.
Also whitelist wheel in tool.uv.exclude-newer-package so the existing
build-system exclude-newer brick guard stays green.
Fixes#96488
Signed-off-by: Olympusbuildz <Olympus.roots@outlook.com>
Co-authored-by: Olympusbuildz <Olympus.roots@outlook.com>
Signed-off-by: Olympusbuildz <Olympus.roots@outlook.com>
archive_and_compact() soft-archives every active row with compacted=1 and
then re-inserts compacted_messages as fresh live rows. When the
compressor's protected tail rides inside that list verbatim - which is
the normal batch-compaction shape ([summary] + tail) - the tail's
ORIGINALS end up stored twice per compaction: (active=0, compacted=1)
next to their live clones. search_messages() recalls both flags without
DISTINCT, so every carried-forward message came back once per compaction
(measured up to 4 identical hits) and was mislabeled to users and the
agent as archived "summarized away" content.
Add an optional tail_count parameter: the last tail_count archived rows
are superseded byte-identical duplicates, stamped rewind-style
(active=0, compacted=0, hidden from recall) instead of compacted=1.
Callers:
- batch in-place compaction counts the compressor-tagged tail dicts
(_COMPACTION_TAIL_MARKER set by compress() on every carried-forward
message);
- micro-compaction splices [prefix, marker, suffix] - everything except
the single marker row is carried forward, so tail_count=len-1;
- proactive tool-result pruning rewrites content in place (not verbatim),
keeping the historical archive-everything behavior.
Fixes#86366
Runs the installer's real node_deps_workspace_args against fabricated
checkout layouts by sourcing install.sh in --manifest mode, which defines
its functions without performing an install.
The load-bearing assertion is the invariant that no checkout shape lets
apps/desktop resolve, including the empty-argument case that would silently
hand npm the whole workspace glob back.
Also cover classify_changes recovering the PR file list when compare
returns nothing, so fail-open does not demand ci-reviewed for a CLI-only
install change.
The browser-tools step ran a bare `npm install` at the repo root, which
resolves the root package.json's `apps/*` workspace glob. That materializes
apps/desktop and with it node-pty, which ships no Linux prebuild and falls
back to `node-gyp rebuild` — so the installer needs make/gcc on a machine
that will never launch Electron or a PTY addon. Since #85297 made a failed
npm install fatal, a host without a C toolchain (a stock CentOS/RHEL box,
for instance) cannot complete a CLI-only install at all; it just reports
"npm install failed or timed out".
Name the workspaces the install actually needs instead. ui-tui and web are
selected when present, with --include-workspace-root so the root's shared
ESLint devDependencies are not pruned by the scoped install — the same
closure `hermes update` already installs. A checkout with neither workspace
falls back to a root-only install, since npm fails hard on a workspace it
cannot find. Desktop dependencies keep coming from install_desktop(), which
is only reachable via --include-desktop.
Against a pristine tree the unscoped install reifies 1362 packages including
node-pty 1.1.0; the scoped one reifies 582 with no native desktop addon.
A fork force-push can 404 the compare API used by detect-changes, which
fail-opens with ci_review=true and blocks the PR on a ci-reviewed label
the install change does not need. Recover the file list from the pull
request files endpoint before that fail-open.
AI-review follow-up on #87467:
- On probe failure, remove the extracted ~/.hermes/node tree and the
node/npm/npx bin links so later installer steps and retry runs start
clean instead of resolving node to a binary that cannot start.
- The termux pkg branch had the same silent-success class: an empty
version probe logged success and set HAS_NODE=true. Degrade with the
binary's own error instead.
install_node's post-install probe was
installed_ver=$(node --version 2>/dev/null) under set -e: when the
downloaded Node exists but cannot start (Node 26 linux-x64 builds link
libatomic.so.1, missing on minimal Debian/Ubuntu), the assignment
aborted the whole installer at exit 127 with the loader's explanation
discarded — installs died mid-sentence with no output at all (#87460).
- Probe now captures stderr and degrades with log_error carrying the
loader message plus the libatomic1 hint instead of aborting.
- Debian/Ubuntu installs preinstall libatomic1 (best-effort, mirroring
the existing apt idiom) so the common case just works.
- Termux branch's same-shaped probe gets a || true guard.
Fixes#87460
Review feedback on this PR: without --no-checkout, the blob fetch runs
inside git clone's own checkout step, so when the repo-scoped 429 hits
that fetch the whole clone exits non-zero, the else branch removes the
directory, and the fallback degrades to one more failed clone under
exactly the condition it exists for.
- Clone with --no-checkout (commits+trees only — small, passes the
throttle); the blobs are then fetched by a separate 'git reset --hard
HEAD' the retry can actually wrap. Verified on a local file://
filtering remote: the no-checkout clone materializes nothing and the
reset alone produces the full working tree.
- Fail closed: both reset attempts failing now removes the checkout and
reports 'Failed to clone repository' instead of the previous '|| true'
+ unconditional clone_ok=true handing the installer a half-materialized
tree printed as a success.
- The reset runs under a subshell cd so a failed materialization never
leaves the shell in a deleted cwd, and the direct-retry loop bound now
derives from $max_attempts (seq) instead of a hardcoded 1 2 3 4 that
could drift from the reported attempt count.
GitHub throttles packfile generation for this repository with
repo-scoped HTTP 429s that are not client IP rate limits: an
anonymous clone of a small repo succeeds and the API quota is
untouched, but the single big pack behind --depth 1 dies
mid-transfer with 'RPC failed; HTTP 429 / expected packfile'. The
fresh-install clone path had no retry and no fallback, so a clean
machine exited 1 at the download stage and left a half-populated
install directory (same throttle as the update path in #89287).
Retry the HTTPS clone with linear backoff, removing the partial
clone between attempts; when every direct attempt fails, degrade
to a blobless partial clone and materialize the working tree with
a hard reset — many small packs instead of one big one, which is
what gets past the throttle. SSH-first ordering, the existing
installation update branch, and the commit-pin flow are unchanged.
Widens the salvaged cron doctor with the highest-value fleet check:
an active job whose next_run_at is parked >15min in the past is not
firing (dead ticker, downed gateway, wedged fire-claim). Also registers
doctor in the docs (cron guide + CLI reference) and resolves the salvage
onto current main alongside runs/incidents/notepad.
The multiplex loop on current main filters profile homes through
_existing_profile_homes (#47368); literal non-existent /tmp paths are
skipped, so the salvaged test's homes must exist on disk.
A long-lived process whose checkout was updated underneath it (hot git
pull, interrupted hermes update) serves mixed sys.modules. When such a
stale process races a fresh gateway for the cron tick lock and wins the
minute, every agent job it dispatches can die on ImportErrors whose real
cause is staleness — and the fresh gateway's ticker skips the same minute
as lock-loser, so the user's scheduled job fires broken or not at all.
tick() now checks, BEFORE acquiring the tick lock:
skew detected (boot fingerprint != disk revision)
AND this process does not own the gateway runtime lock
AND that lock is held (a fresh gateway is alive)
-> raise CronTickYielded, skipping the tick entirely
Each arm alone keeps the old behavior:
- skew + self-owned lock -> proceed (delivery-path stale-code hint stays
the surface for gateway-owned dispatches)
- skew + no lock holder -> proceed (desktop-standalone users must not
lose their only ticker to a silent yield)
- skew None (non-git install, no boot fingerprint, probe failure) ->
proceed; yielding is a certainty claim, never a guess
The yield RAISES instead of returning 0 so the provider loops record it
via record_ticker_error and mark the heartbeat success=False — a yielded
tick must not look like a healthy one (hermes cron status shows why),
mirroring the EMFILE propagation contract (#87644). Yield logging is
throttled to once per skew episode. Self-healing: when the fresh gateway
dies, its lock releases and the stale ticker's next tick proceeds.
Multiplex loop: a yield for one profile no longer cancels sibling
profiles' ticks in the same cycle; only the yielding profile records an
unsuccessful beat.
gateway/status.py gains owns_gateway_runtime_lock() —
is_gateway_runtime_lock_active() is True for the lock's own owner too, so
a caller deciding whether to yield to a FRESH gateway needs the
in-process handle as the discriminator.
Follow-up on the cherry-picked #92489 base: replace the four separate
process-global cache slots with one lock-guarded identity->(name, zone)
mapping so racing profile-scoped threads can never publish a mixed
identity/value pair (the P1 interleaving flagged in the #92489 review),
keep each profile's resolved zone hot across multiplex switches, and pin
the #97905 symptom with a real-store regression test: a foreign-process
tick (desktop multiplex ticker pattern) must persist next_run_at with the
job-owning profile's UTC offset.
Fixes#97905. Refs #88220, #92489.
On a fresh boot with no resume_pending sessions, _finish_startup_restore
opened the inbound gate almost immediately while the agent-side turn
machinery (run_agent import graph, tool schemas + check_fn probes,
context-file tier) was still cold. A message arriving in that window was
served with a skeleton system prompt (~1.7K tokens vs ~14.6K healthy):
no AGENTS.md/context tier, no tool schemas, memory provider initializing
mid-turn.
Fix: start a background turn-machinery warm-up when the startup gate
closes (overlapping the network-bound platform connects) and have
_finish_startup_restore await it — BOUNDED by
agent.gateway_startup_warmup_timeout (default 20s, 0 disables) — before
draining the queue and opening the gate. On timeout the gate opens
anyway and the warm-up finishes in the background, so a wedged init can
never make the gateway permanently unavailable.
Reported by @yhfmstr in #99373.
Fixes#99373
TUI server shutdown stamps ended_at/end_reason='tui_shutdown' on sessions
whose agent keeps running; every rotation then aborts at
publish_compression_child's liveness check forever (the #88197 wedge; the
amplification half was fixed by #88411).
Class fix: is_automatic_end_reason() in hermes_state_common owns the
"accidental infrastructure cleanup vs deliberate boundary" taxonomy.
publish_compression_child clears automatic stamps in its own transaction
and proceeds (parent re-closes with its TRUE boundary,
end_reason='compression'); the #88411 pre-flush guard no longer aborts on
stamps the publish can heal. Deliberate boundaries (compression,
session_reset, explicit close) still fail closed at both sites.
TEST REPIN (deliberate contract change):
test_ended_parent_aborts_before_the_prepublish_flush pinned
"tui_shutdown stamp => rotation aborts and parent must not grow" — the
abort it required IS the #88197 wedge. Repinned as two tests:
- test_automatic_stamp_no_longer_wedges_rotation: automatic stamp =>
rotation COMMITS (no abort loop, so no growth-by-abort is possible);
- test_deliberately_ended_parent_aborts_before_the_prepublish_flush:
session_reset (deliberate boundary) => still aborts BEFORE the #47202
flush, preserving #88411's no-growth contract where an abort remains
correct.
The class invariant "no aborted rotation grows the parent" holds
everywhere: automatic stamps no longer produce aborts, deliberate
boundaries still abort pre-flush.
Hardening on top of @Soju06's forwarding fix: v2 providers written against
the original docs example (def on_pre_compress(self, messages)) must not
TypeError when the host forwards require_checkpoint — inspect the signature
and fall back to the legacy call shape. Docs example updated to advertise
the keyword.
MemoryManager.on_pre_compress() detects checkpoint API v2 providers,
selects the normalized evidence list for them, and re-raises their
failures under require_checkpoint — but it never tells the provider
that a checkpoint is required: the call passes only the messages.
A v2 provider therefore runs in its default best-effort mode, swallows
durable-write failures, and returns normally; the host then treats the
checkpoint as succeeded and lossy compression proceeds. With
compression.checkpoint_required: true this silently defeats the
guarantee the option exists to provide.
Forward require_checkpoint only to providers advertising the requested
checkpoint API version. Legacy providers keep the strict one-argument
on_pre_compress(self, messages) contract, so bundled v1 providers
(honcho, mem0, supermemory, ...) are unaffected.
Regression tests cover required and best-effort signaling, legacy
signature compatibility, and required-mode failure propagation.
Prove a fence-cancelled helper (no abort flag) persists cooldown so the
next turn does not re-arm compression, the host does not wait out the
600s ceiling after cancel, a held lock skips the sibling agent, and
unwind cancellation records the same brake.
(cherry picked from commit d2e178cd96fbcc8b2baf68488a8f46c70bff31a2)
A /stop or /restart abort left hygiene with no cooldown, so the next turn
re-armed auto-compression and waited up to 600s behind a fence that would
refuse the commit again. Record a cooldown on fence-cancel and unwind,
stop extending that wait once the fence is cancelled, and skip a new
hygiene agent while a compression lock is already held.
Pin gateway re-tagging of idle/preflight lifecycle lines as compacting,
and assert the TUI keeps that status until compacted rather than
restoring the busy bar after 4s.
Idle and preflight compaction arrived as lifecycle status without the
"Compacting context" marker, so TUI never entered a compacting state.
Re-tag those lines and freeze the busy FaceTicker on "compacting" for
the whole pause instead of restoring "running…" after 4s.
Re-applied onto 3aee29089 after `hermes update` reset main to origin/main.
1. web_routers/sessions.py: _resolve_session_id() classifies malformed-DB
errors via the existing is_malformed_db_error() and raises 503 at all five
call sites. delete_session_endpoint was the worst — an unresolvable id
counted as idempotent success, so DELETE reported it had removed a session
that was still on disk.
2. gateway/lifecycle_ledger.py: check_state_db_integrity() runs PRAGMA
quick_check(1) on the unclean-exit path only (~2s on 500MB) and records the
verdict into gateway-exit-diag.log. The 2026-08-31 corruption sat undetected
for 3.5 days because nothing ever looked.
3. hermes_cli/gateway.py: `gateway run --replace` gave the outgoing gateway 5s
before SIGKILL; SessionDB.close() runs a PASSIVE WAL checkpoint that does
not finish in 5s on a WAL 4x past the autocheckpoint threshold, and a kill
mid-checkpoint tears b-tree pages. Grace raised to 30s via a testable
_await_gateway_exit() that also re-checks after the final sleep (a PID
exiting in the last interval must not be SIGKILLed — PID-reuse hazard).
NOT added: wal_checkpoint(TRUNCATE) at shutdown — removed upstream in #45383
because a TRUNCATE reset races the live writer and tears b-tree pages.
Adversarial review: Codex gpt-5.6-sol, 9.0/10 across three groups, no must-fix.
The salvaged #98948 change returned False for every db=None, which also fired in deliberately store-less/degraded contexts (no _db_error), regressing six prompt.submit tests. Gate the loud failure on _db_error being set — the actual #98924 symptom — and keep the pinned best-effort contract otherwise.
Follow-ups on the salvage: regular-file guard before the zeroed byte-probe (a FIFO at the state.db path would block startup forever — #98017 review P2), plus an on-main-reproducing UnicodeDecodeError fixture for #98924 (raw bytes in sqlite_master, not messages.content, are what reach pysqlite error-message decode).
Companion to #98935, which fixes _fts_table_probe itself. This covers the
surfaces that PR does not touch:
- web_server._open_session_db_at_path: the one-writable-open heal only
caught sqlite3.DatabaseError; a raw UnicodeDecodeError (pysqlite failing
to decode SQLite's own error message over corrupt file bytes) bypassed
it, so the heal documented for malformed schema never fired (#98924
Failure 1). Both catches widened; decode errors dispatch to the heal.
- SessionSchemaMixin._recover_stale_fts_locked: drop-and-recreate skipped
vtables whose probe raised UnicodeDecodeError, the same too-narrow
catch the issue identified in the probe.
- TUI gateway: _ensure_session_db_row returned silently when the store
could not open, so prompt.submit streamed the turn while persisting
nothing (#98924 Failure 2). It now returns False and prompt.submit
fails the RPC with code 5072 so desktop maps it to a toast, mirroring
the disk-full/5070 convention. session.create stays silent per its
pinned degraded-mode contract.
The tokenizer-not-loaded branch ran its sqlite_master presence check
and self-heal statements (drop stale cjk triggers) with no guard, so a
transient sqlite3.OperationalError there (e.g. a locked database)
escaped the method despite its documented "Never raises" contract.
That exception then hit _migrate_broad_fts_update_triggers's
quarantine-then-reraise handler, aborting _init_schema and the whole
SessionDB open. Wrap the presence-check/self-heal block in the same
kind of OperationalError guard the tokenizer-loaded branch already
has, degrading to "no cjk index" instead of propagating.
Invalid UTF-8 bytes in messages.content (e.g. 0x81 from hardware issues,
corrupted disk I/O, or manual DB edits) caused read-only SessionDB init
to die on a bare UnicodeDecodeError in _fts_table_probe, taking down
every read endpoint (GET /api/sessions, Desktop read-only opens of other
profiles' DBs). The probe caught only sqlite3.OperationalError and would
re-raise any other exception, including UnicodeDecodeError (a ValueError,
not an sqlite3.Error subclass).
On some Python/SQLite builds the decode failure surfaces as
UnicodeDecodeError; on others as OperationalError('Could not decode to
UTF-8 column ...'). The fix catches both and treats them the same:
the FTS index is degraded (search may return less or fail), but the store
itself stays accessible for writes and non-FTS reads. Writable init
schedules a rebuild or degrades to LIKE search until repaired.
Adds test_98924_readonly_fts_decode_error.py with a regression test that
injects invalid UTF-8 via CAST(x'...' AS TEXT) through the Python sqlite3
module, triggers an FTS rebuild, then confirms that read-only init succeeds
instead of raising.
- Guard against concurrent-opener race where newly created 0-byte state.db was falsely quarantined before first schema write
- Wrap startup in quarantine_cross_process_lock when database is uninitialized or zeroed
- Guard is_zeroed_sqlite_file and is_zeroed_state_db against active live connections in current process
- Add concurrent-opener and live-connection regression tests
Salvaged from PR #90734 by @Kyzcreig onto current main:
- hermes_state_search.py: post-commit FTS incremental merge failures
(including the bare SystemError CPython's sqlite3 layer raises under
cross-thread errmsg scrambling) are contained and logged instead of
escaping and making the caller replay an ambiguous, possibly-durable
write (exactly-once refinement by @yuzilongleif-collab).
- tests/state/test_writer_conn_thread_safety.py: live reader/writer race
hammer + AST sweep freezing the no-unlocked-writer-conn invariant.
On top: the sweep now also flags self._conn PASSED to helpers, which
caught one more live site on main — get_session_delete_targets handed
the shared writer connection to _collect_delegate_child_ids inside a
_read_ctx block, executing on it without self._lock. Routed to the
borrowed read connection.
Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
Co-authored-by: yuzilongleif-collab <235949691+yuzilongleif-collab@users.noreply.github.com>
Salvaged from PR #99435. Fails on the next lock-free self._conn.<method>()
call site added to hermes_state.py; reads belong in _read_ctx(), writes
under self._lock (#99349).
Co-authored-by: dimitrysuen <dimitrysuen@users.noreply.github.com>
- fresh conversation subscribes with no since floor + bounded limit; seeded
channels keep since=last_ts-1 (#78429)
- all three production CLOSED membership phrasings prune per-subscription
without reconnect (#76850 + #97502)
- restricted channels are not re-adopted by discovery sweeps
- WS periodic discovery subscribes conversations found without a membership
event (#93557) and the companion task dies with its connection
- watch-all live channel adoption vs explicit-list scoping (#75107)
All six new tests sabotage-checked: each fails when its fix is reverted.
Three sibling gaps in the WebSocket transport's conversation lifecycle:
- #78429: _send_channel_subscription defaulted a zero last_ts to
'since ~ now', so the message that CREATED a new conversation (created_at
fractionally before the subscription) was never delivered. A channel with
no high-water mark now subscribes from the beginning with
limit=_FETCH_LIMIT instead; seeded channels still resume from last_ts-1.
- #93557: relays do not guarantee a kind-44100 membership event per new
conversation, so WS-transport deployments never discovered DMs opened
mid-session until a reconnect. The WS loop now runs the same
_discover_dms sweep the poll transport uses, on the same cadence
(poll_interval * _DM_DISCOVERY_EVERY), via a companion task that is
cancelled with the connection.
- #75107: _discover_dms only ever adopted DM-shaped conversations, so a
real community channel the agent joined mid-run was never subscribed
until restart. In watch-all mode (no explicit channels list) newly
listed real channels are now adopted and seeded from their newest events
(history predating the join is not replayed). Explicit watch lists stay
authoritative.
- Widen the permanent-rejection match to the exact relay phrasings seen in
production (#97502): 'not a channel member' and 'auth-required', alongside
'restricted'.
- Close the re-adoption hole called out in review: _discover_dms() (both the
dms-list path and the channels-list fallback) now skips channels in
_restricted_channels, so a restricted channel dropped at runtime cannot be
silently re-added by the next discovery sweep and re-trigger the rejection.
- Credit: runtime CLOSED matching terms from PR #97502 by @repfigit; the
per-subscription drop + restricted set is PR #76850 by @xozai.
When a Buzz relay sends a CLOSED frame for a single subscription with a
'restricted: not a channel member' error, the adapter was raising
ConnectionError, tearing down the entire WebSocket connection, and
immediately reconnecting — causing a ~1.6 s flood in gateway.log.
Root cause: the CLOSED handler unconditionally raised ConnectionError
regardless of whether the error was permanent (restricted) or transient
(e.g. server shutdown).
Fix:
- On a 'restricted' CLOSED, drop only the offending subscription and
record the channel in a new _restricted_channels set instead of
tearing down the whole connection.
- Skip restricted channels during connect() seeding and
_subscribe_websocket() so reconnects don't re-trigger the same error.
- Non-restricted CLOSED frames still raise ConnectionError and reconnect
as before.
Adds three regression tests:
- test_websocket_loop_drops_restricted_channel_without_reconnect
- test_websocket_loop_reconnects_on_non_restricted_closed
- test_restricted_channels_skipped_during_subscribe
Tested on macOS against buzz.xozai.com: gateway.log shows zero
'restricted' errors and stable 'watching N channel(s) via websocket'
after the fix.
walker