get_messages_as_conversation, get_resume_conversations, and
get_ancestor_display_prefix still took self._lock — the same global
choke point the read-path split (WAL per-thread read-only connections)
was meant to remove from every recall/browse read. These three are the
hottest reads in the file: every session resume across the gateway,
CLI, and ACP adapter goes through one of them, so a resume racing a
burst of concurrent-session writer flushes still convoys behind them
exactly like the fixed paths used to.
_session_lineage_root_to_tip (the lineage walk shared by all three,
plus get_conversation_root) had its own independent self._lock use and
needed the same conversion — without it the outer functions still
blocked on the very first line.
Verified empirically: a reader thread calling all three functions
while another thread holds self._lock blocked for the writer's full
hold duration before the fix, and returned immediately after (SQLite
3.50.4 in this dev venv falls back to journal_mode=DELETE per the
WAL-reset-bug guard, so the requires_wal-marked regression test is
exercised via a local WAL-forced script instead; it still runs and
passes on any runtime where WAL is actually active).
Re-derivation of #38491 by @stremtec onto current main (the original is
10,119 commits behind; the hook moved into ui-tui/src/app/). The hook
returned a fresh object literal every render, defeating memoization in
useMainApp's consumers; useMemo over the (all-useCallback-stable)
handles makes the return referentially stable.
Dep array covers ALL nine returned handles incl. trimTail (the
re-derivation initially omitted it - stale-closure class).
CI-caught: test_verification_stop_caching and test_tui_gateway_server::test_native_vision_turn_persists_a_renderable_image_ref both assert on append_message.call_args, but the flush loop now calls append_messages_batch. Same class of test-fake fallout fixed in 5 other files — these two were missed.
Simplify-pass folds on the #23254 salvage:
- REUSE (HIGH): append_messages_batch now delegates row serialization to
the pre-existing _insert_message_rows helper (already shared by
replace_messages / archive_and_compact / portability import) instead
of adding a third serialization path (_prepare_message_row +
_MESSAGE_INSERT_SQL are gone). One row-writer for every multi-row
path; the row-ID return was consumed by no production caller, so the
batch returns the inserted count.
- QUALITY (HIGH): the compression-lock + compression-closed admission
guards are extracted into _check_transcript_write_guards, shared by
append_message and append_messages_batch (previously duplicated 23
lines that had already needed targeted fixes, #74478). The role-gated
reasoning filtering is no longer duplicated in run_agent.py — it
lives at its one site inside _insert_message_rows.
- EFFICIENCY (MEDIUM, measured): unbounded seed copies hold one BEGIN
IMMEDIATE for seconds (10k rows ~= 2.4s; FTS triggers dominate) and
monopolize the in-process write lock. append_messages_batch grows a
chunk_rows param; all seed/copy call sites use chunk_rows=500. Same
recovery semantics as the old per-row loops, bounded lock holds.
- REUSE (MEDIUM): the two remaining per-row branch-copy loops found by
the pass (gateway/slash_commands.py /branch, hermes_cli
cli_commands_mixin.py branch) are converted to chunked batches too
(AsyncSessionDB's generic to_thread forwarder covers the async site).
Turn-flush benchmark unchanged after the refactor: 2.43 -> 0.87 ms
median per 5-message flush (64% faster).
The flush now goes through append_messages_batch; MagicMock-based
assertions and barrier fakes that hooked append_message observed
nothing (the flush's try/except swallowed the AttributeError). Assert
on the batch payload instead.
Sibling sites of the per-message flush pattern: both branch-seed
paths (session.branch in methods_session.py and the lazy seed persist
in server.py) copied the parent history row-by-row -- one transaction
per row, and a branch seed can be hundreds of rows. Route both through
SessionDB.append_messages_batch. The server.py path also gains real
atomicity: _branch_seed_persisted assumed every row landed, which the
per-row loop could not guarantee.
Re-derivation of #23254 (@devsart95) on today's flush loop. The turn
flush in _flush_messages_to_session_db wrote one BEGIN IMMEDIATE
transaction per message row; a typical agent turn (user + assistant +
tool results) paid 3-8 transactions -- and, off WAL (the default on
macOS while the WAL-reset guard is active), 3-8 fsyncs -- per turn.
Adds SessionDB.append_messages_batch: same row shape as append_message
(shared _prepare_message_row serializer + _MESSAGE_INSERT_SQL column
list, so the two writers cannot drift), same compression-lock and
compression-closed guards, one aggregated session-counter UPDATE, one
transaction for the whole batch. Row serialization stays outside the
write lock.
The flush loop now collects the turn's new rows and writes them in one
call. All-or-nothing pairs exactly with the persisted-marker stamping:
on failure no rows landed and no markers were stamped, so the next
flush re-writes the whole tail (same recovery contract as before,
minus the partial-prefix case that could double-count).
Measured (same harness, 5-message turn, journal_mode=DELETE,
synchronous=FULL): 2.32ms -> 0.83ms median per turn flush (64% faster,
5 fsyncs -> 1). On WAL the win is smaller but the atomicity fix holds.
SubdirectoryHintTracker re-injected identical context files whenever the same
AGENTS.md was reachable through more than one path. Symlinked shared
workspaces, hardlinks, and timestamped backup copies all alias a single file,
so a normal session could ship the same 8KB of instructions two or three
times. Nothing deduped it and nothing excluded directories that only ever
hold copies.
Two changes:
* Track a sha256 of every injected hint body. Repeat content is skipped, and
the working directory's own context file is seeded at construction so the
copy prompt_builder already loaded at startup is never sent again.
* Skip directories that hold copies rather than authoritative context
(backups, node_modules, venv, site-packages, .git, .Trash, vendor, caches).
Screening is relative to working_dir, so a project that legitimately lives
under vendor/ keeps discovering its own subdirectory hints.
Measured on a real session that touched a symlinked shared workspace:
3 injections / ~24,000 chars before, 1 injection / 8,112 chars after.
14 new tests cover symlink aliasing, byte-identical copies, working-dir
seeding, distinct content still being injected, each excluded directory name,
excluded ancestors, and the working-dir-inside-excluded-name case.
Review follow-up for salvaged PR #76782. Three setup-wizard
validation functions called _normalize_openviking_url outside their
try/except blocks. Since _normalize_openviking_url now raises
_OpenVikingEndpointError for blocked or malformed endpoints, an
invalid endpoint would crash the wizard instead of returning a
friendly (False, message) tuple.
- _validate_openviking_auth: move _normalize_openviking_url inside try
- _validate_openviking_root_access: same
- _validate_openviking_setup_values: catch _OpenVikingEndpointError explicitly
- Remove dead ternary in _normalize_openviking_url safety check (candidate
always has http/https scheme by that point)
- Replace redundant float('-inf') < x < float('inf') with math.isfinite()
in _setting_float; drop the redundant infinity check from _setting_int
(is_integer() already rejects inf/nan)
_recall_config() previously read all settings (recall_limit, score_threshold,
recall_resources, etc.) exclusively from environment variables. This forced
users to store behavioural configuration in .env, violating the Hermes
convention that .env is for secrets only.
The infrastructure to load config.yaml -> memory.openviking was already in
place via _load_hermes_openviking_config(), but _recall_config() never
called it.
Fix: call _load_hermes_openviking_config() and pass its values as the
default parameter to _env_int/_env_float/_env_bool. Env vars still override
config.yaml values, preserving backward compatibility.
Closes#62540
(cherry picked from commit 6aadf1256835745e0302aa3d3b5ae0660b368637)
OpenViking is_available() only consulted env vars and use_ovcli_config, so an
endpoint saved to config.yaml (e.g. by the Dashboard) reported needs_config;
_resolve_connection_settings() likewise never folded config.yaml's non-secret
fields into its chain. RetainDB initialize() read base_url/project from the
environment only, ignoring the values the Dashboard writes to config.yaml.
Both now resolve non-secret fields as env -> (ovcli ->) config.yaml -> default;
secrets still come from the environment. Adds regression tests for both.
Fixes#68209
(cherry picked from commit dca57915b97b5705b30927a062e1d0f2f23d3841)
Review feedback: the previous test called _mark_session_committed
directly, so it verified the guard's behavior but not the wiring that
sets it — a future break in the commit_memory_session -> same-id
compression-boundary path would not be caught.
Add a lifecycle regression that drives the real sequence: on_session_end
commits through the actual path, on_session_switch(same id,
reason="compression") crosses the boundary, sync_turn records a genuinely
new turn, and a second on_session_end must produce a second commit POST.
Without the fix it fails showing exactly one commit call, which is the
reported data loss: every turn after the first compression is dropped.
The rotation and /undo tests stay as scope guards.
(cherry picked from commit 0ca5a330630a30b105cbbc32e8a23f2c5ffe0eab)
`_committed_session_ids` is a permanent per-sid latch, and
`_session_needs_commit` checks it before the turn counter by design — a
racing sync_turn can re-increment `_turn_count` after commit+reset, so
the guard must win to stop a double-commit.
That is correct for a session being left behind. It is wrong for one
that keeps its id. `compress_context()` commits before rewriting the
transcript in both modes, and with `compression.in_place: true` (the
default) `on_session_switch` receives the same id and does not rotate.
The latch then rejects every later commit for a still-live session — the
next compression, /new, normal session end, startup recovery — so every
post-compression turn is silently never extracted.
Rotation mode is unaffected because a fresh child id is minted and
starts clean, which is what confirms the latch's intent was only ever to
dedupe the departing id.
Clear the latch when compression completes without rotation. Turns
arriving after that point are genuinely new, and this is a defined
moment rather than a race. The rotation path is untouched, so the old
id stays latched and its _finalize_session_async still dedupes against
the compression commit.
Fixes#74695
(cherry picked from commit d1e5c3dc33ef0d43d021662674e1a7cd5e43eecd)
The provider used to disable OpenViking permanently when the server was
unreachable. That was fixed: `_ensure_client()` now reconnects lazily,
with a 30s cooldown gate in `_ensure_client_locked`.
Only one of the seven user-facing warnings was updated to match. The
other six still told the user memory was "disabled for this Hermes run",
which is no longer true — every one of those paths is retried on the next
access. A user who reads the old message has no reason to retry, which is
very likely how #5721 ("never recovers") came to be filed against
behaviour that already recovers.
All six sites were traced to confirm none is terminal for the run: the
`initialize()`-time and waiter-thread failures never arm `_failed_refresh`
(only line 2439 does), so they retry on the very next access with no
cooldown at all.
The replacement wording deliberately omits the "(after cooldown)"
parenthetical used at the already-correct site — that detail is only
accurate where `_failed_refresh` was just armed. The neutral phrasing is
true at all six.
Also promotes two clause separators to periods to avoid "…; …disabled;"
collisions.
(cherry picked from commit 8346403a4b97af503d26b0f7905ff513828d821e)
`_start_local_openviking_server()` spawned `openviking-server`
unconditionally. Both callers — `initialize()` and the runtime
unreachable handler — reach it from a health probe, and that probe can
time out client-side while the server is up and serving. The spawned
process then loses the data-directory lock and exits immediately with
`DataDirectoryLocked`; because the probe keeps timing out, the cycle
repeats every cooldown window (~5 min observed).
The existing 30s `_failed_refresh` cooldown paces the loop but cannot
stop it, since it expires while the underlying condition persists.
Probe the target host:port before spawning and treat an occupied port as
already-started. This guards both call sites at their single convergence
point. The probe deliberately tests only that a listener owns the port —
enough to know a second server would lose the lock — and says nothing
about that listener's health.
The parse/probe now precedes the PATH lookup, so a reachable server is
reported as running even when `openviking-server` is not on PATH.
Fixes#74846
(cherry picked from commit b49427d85fd6628eb4a7fe099e5c390c5c4cc935)
## Summary
- Normalize OpenViking endpoints through `is_always_blocked_url` and fall back to the default local endpoint when poisoned.
- Keep intentional loopback / LAN self-host working.
- Add focused unit tests.
## Salvage / credit
Memory-provider endpoint floor sibling of RetainDB/Supermemory always-blocked hardening (avoids over-broad #4984-style private-IP bans).
(cherry picked from commit 8fa607d0aedb8c5fca398d7f112b1b25ade54fa2)
Simplify-pass fold: to_drop names come from the literal update_names\nallowlist via IN binding, so the [A-Za-z0-9_]+ fullmatch could never\nfail — and if it somehow did, its `continue` would miscount (the\nskipped trigger stayed in len(to_drop)/the log while CREATE TRIGGER\nIF NOT EXISTS silently kept the broad variant). Delete the guard and\nits function-local re import; keep the invariant as a comment.
_ensure_fts_cjk_schema never raises on OperationalError; post-condition
after dropping messages_fts_cjk_update now requires a narrowed UPDATE
trigger or durable fts_cjk_stale + unavailable. Covers the production
soft-fail path the raise-only handler missed.
Retarget #73639 onto the SessionDB mixin split (hermes_state_common /
hermes_state_schema). Fresh installs create UPDATE OF content/tool_*
triggers; existing broad AFTER UPDATE triggers are inspected and
replaced under schema init without an FTS rebuild (WHEN clauses already
guarded content correctness; OF skips non-content status writes that
saturated disk I/O on large state.db).
Tests: tests/test_fts_update_of_narrowing.py (4)
CI-caught: cron-jobs-section had an extra blank line between sorted imports; use-message-stream's visibility-flush effect assigns flushHandleRef.current=null inside a useEffect (legitimate timer-clear, not an atom mirror) — eslint-disable-next-line per the rule's documented convention.
Two findings from the simplify pass on the final trio diff:
- status-pulse: one pause controller + one aligned period timer shared by all StatusPulse instances (ref-counted), instead of N x (document/window/bridge listeners + unsynchronized 5s wakes) — a sidebar can show dozens of pulsing dots. Pause still cancels in-flight animations so the compositor sleeps immediately.
- cron-jobs-section: the runs-peek effect created its interval even while the pane was hidden (callback no-oped but the timer still woke the renderer every 8s/60s per expanded job). Early-return when hidden — visibility is already in the dep array, so becoming visible restarts load + timer.
Partial pick of the surviving renderer hunks from #75395 (perf commit
6502e441d plus fixup 3fbbc9c1d): gate the 500ms subagent now-ticker and
the cron sidebar 1s ticker/run-poll on usePaneVisible, and skip the
legacy floating-pet poll while the document is hidden. Dropped hunks
(electron/main.ts, vitest.setup.ts/config) intentionally excluded.
E2E guard for the salvaged PR #71755: database.cache_size/mmap_size/
temp_store from config.yaml must reach the writer connection, the
read-only cross-profile attach, and the WAL per-thread reader — and a
default install (no database: keys) must keep byte-identical SQLite
defaults on every connection type. Also covers integer-coercion
rejection of garbage values for the three new keys.
cache_size uses -16000 (not the doc example -2000) because -2000 is
SQLite's compiled-in default and would not discriminate a regression.
Addresses review from @teknium1 on PR #71755:
- Extended apply_database_pragmas() to handle cache_size, mmap_size,
and temp_store from config.yaml (alongside existing wal_autocheckpoint
and journal_size_limit). No hardcoded defaults — all values are
opt-in via config.yaml, avoiding policy conflicts with other PRs.
- Applied to ALL connection types: writer (_connect_and_init),
read_only cross-profile attach, and WAL per-thread readers
(_get_read_conn). Previously PRAGMAs only ran on the writer path.
- Removed inline PRAGMAs from _connect_and_init — single source of
truth in apply_database_pragmas().
- Documented config keys with examples in function docstring.
Cross-PR interaction fix: #77714 (salvage of #71775) changed
_available_entries to return (available, pending_refresh) while #77631
(salvage of #67642) added next_available_at() which still truthiness-
tests the bare return. A non-empty tuple is always truthy — even
([], []) — so the reset-aware gate silently returned None ('no wait
info') for every exhausted pool, disabling the feature #77631 shipped.
Unpack the tuple and test the available list.
Also adapts the lock-probe test for the RLock introduced by #77714
(same-thread non-blocking acquire always succeeds on an RLock; probe
from a helper thread instead).
Simplify-pass finding: _listing_group_label already falls back to 'other' for empty source names, and _classify_source guarantees source_name=='' only when source=='other' — both legs were dead by construction. Aligns the summary path's grouping with the listing path.
kshitijk4poor