Efficiency-pass follow-up on the #66355 salvage: force=True bypassed the
cooldown entirely, and AIAgent.close() fires a forced trim for EVERY
in-process child subagent close (delegate_tool child.close(), parent
close step 5). A delegate batch of N children closing back-to-back in
the gateway process stacked N+1 uncooled full gc.collect()+malloc_trim
passes (50-500ms each with a large live heap). Forced trims now honor a
5s floor — bursts coalesce, the parent's final close-trim still fires.
Guard test mutation-checked (floor zeroed -> test fails).
Simplify-pass follow-up on the #66355 salvage:
1. _config_settings runs on EVERY trim attempt (before the cooldown
check) and only reads — swap load_config for load_config_readonly.
Deep-copying the whole config per attempt generates exactly the
allocator garbage this module exists to release. Tests re-seamed.
2. Trim-failure logs demoted warning->debug at all 3 periodic sites
(gateway housekeeping, idle reaper, slash worker): sibling failure
branches in the same loops log at debug, and a persistent failure
(e.g. broken import after a partial update) would otherwise warn
every 60s forever.
3. The frame-inspection test now asserts the expected locals exist
before reading them — a rename in _run_prompt_submit fails the test
loudly instead of vacuously passing on None.
Add config-driven glibc malloc_trim for long-lived Hermes processes:
- hermes_cli/mem_trim.py: trim_memory() with configurable cooldown,
RSS snapshot telemetry, and forced-trim INFO logging
- gateway/run.py: periodic trim in gateway housekeeping loop
- tui_gateway/server.py: trim in idle reaper (~every 5 min)
- tui_gateway/slash_worker.py: trim on turn boundary
- run_agent.py: force trim on agent close
- hermes_cli/config.py: context.memory_trim config section
(enabled, cooldown_seconds, log_every_n, info_log_min_delta_mb)
CSA tier-4 reviewed (4 rounds, 0 HIGH/MEDIUM/CRITICAL remaining).
Supersedes PR #63708 + #64591 with enhanced telemetry and gateway/slash_worker coverage.
The detached meet_bot child inherits the process environment, not the parent's contextvar secret scope, so process_manager.start() now resolves HERMES_MEET_REALTIME_KEY/OPENAI_API_KEY through get_secret in the parent and passes it explicitly in the child env (spawn-wrap shape). Adds the consolidated per-family regression test file.
Review follow-up on the #62871 salvage (simplify pass, HIGH):
1. Ops unresolved at the wait deadline were RETAINED in the pending set.
A permanently failing status endpoint (auth error, endless 500s, or a
server that loses ops without 404) would grow the set forever and make
EVERY later prefetch burn the full 10s budget re-polling it — and
prefetch()'s bounded 3s join sits on the reply path, so that money-quote
'adds no response latency' claim breaks. Timed-out ops are now dropped
(identical degradation to prefetch_waits_for_retain=False: possibly
stale recall) with a WARNING so persistent server trouble is visible.
Guard test mutation-checked (fails with eviction disabled).
2. Status polls now spaced 0.5s (was 0.05s shared with the local drain
poll): a wedged op cost up to ~200 get_operation_status round trips
per prefetch; now ~20 max over the default 10s budget.
Address PR #62871 review: with the default retain_async=True, aretain_batch
returns when the write is accepted, not when it's durable/recall-visible, so
draining the local writer queue (task_done) is not a read-after-write signal.
The next-turn prefetch could still recall before the just-completed turn was
observable on the server.
- Track the async operation_id/operation_ids returned by aretain_batch
- _wait_for_retains_drained now applies two ordered, budget-bounded barriers:
(1) local writer queue drains, then (2) tracked server-side async ops report
completion via operations.get_operation_status (an explicit read-after-write
condition). NotFound (completed+evicted) counts as done; transient errors
keep waiting until the deadline
- Completed ops are removed from the pending set so later prefetches don't
re-poll them; the whole wait stays off the reply path
- Add TestPrefetchServerRetainVisibility: op-id tracking (single/multiple),
no-op tracking when retain_async=False, prefetch waiting for server
completion before recall, timeout fallback on a wedged op, and NotFound /
transient-error status handling
Async retain already keeps the memory WRITE off the reply path (writes drain
on the single writer thread while the user gets their response immediately).
This closes the remaining retain/prefetch race: the next turn's warm prefetch
runs on its own thread and could recall BEFORE the just-enqueued retain write
lands, silently dropping the latest turn from recall.
- The background prefetch now waits (bounded) for pending retains to drain
before recalling, so warmed context includes the just-completed turn.
- The wait runs only on the background prefetch thread, never the reply path,
so it adds zero latency to the user's response and loses no writes.
- Bounded by prefetch_retain_drain_timeout (default 10s) and polls
unfinished_tasks so a wedged write can't hang the prefetch.
- New config keys: prefetch_waits_for_retain (default true),
prefetch_retain_drain_timeout (default 10.0).
When hrr_dim=1 the prefixed float32 blob (4+4=8 bytes) collides in
size with a raw float64 blob (1×8=8 bytes), making the format
discriminator in bytes_to_phases ambiguous — a legacy blob starting
with HRR1 would be misread as a prefixed float32 vector.
- phases_to_bytes now accepts an optional dim and falls back to
writing raw float64 when the two blob sizes are equal.
- bytes_to_phases prefers the legacy float64 interpretation when
sizes collide and dim is provided, since phases_to_bytes never
writes a prefixed float32 blob at dim=1.
- Three regression tests cover dim=1 write, round-trip, and the
legacy-prefix collision case.
Addresses hermes-sweeper review on PR #30499.
The steady-state turn only bumps updated_at/last_prompt_tokens on one
routing entry, but persisted it through the full index rewrite twice
per turn (get_or_create_session's healthy-path bump + update_session):
every entry re-serialized, DELETE+INSERT of every gateway_routing row,
and a multi-MB sessions.json dump+fsync — ~50ms p50 at ~1,100 routing
keys in production, out of ~175ms total per-turn gateway persistence.
Metadata-only saves now UPSERT the single row via the existing
HermesDB.save_gateway_routing_entry (<1ms). Structural transitions
(create/recover/reset/switch/prune, compression-tip heals) keep the
full rewrite, which also refreshes the legacy sessions.json mirror.
Correctness: each fast save allocates a per-entry revision from the
routing generation counter under _lock, so fast and full snapshots are
totally ordered by number. Under _save_lock the UPSERT is skipped when
a newer full snapshot or a newer fast save of the same key has already
persisted, and a delayed full rewrite folds in fast records serialized
after its snapshot before writing — an older snapshot can never
overwrite a newer one, in either direction. update_session snapshots
peer fields under _lock so a concurrent reset cannot record a torn
peer row; no DB or a failed UPSERT falls back to the full rewrite so
DB-less installs keep sessions.json durable every turn.
Exercises representative migrated sites per cluster: scoped value wins,
scoped miss no-borrow under multiplex, and unscoped-multiplex behavior per
pattern (in-turn get_secret vs Slack-pattern startup fallback).
alongside every set_hermes_home_override() call site on the tui_gateway
path, and converted session.create, session.resume (both the _make_agent
and _init_session scopes), the lazy _build resume and the per-turn submit
handler. session.branch was missed.
The branch handler already binds the parent's HERMES_HOME and its
profile-scoped state.db — its own comment says it mirrors
session.create/resume — but builds the branched agent with no secret scope.
get_secret() then falls through to process os.environ, which in an
app-global/remote backend is the LAUNCH profile's environment: a session
branched off profile X authenticates with profile Y's credentials. That is
the same cross-profile resolution #67605 fixed for the sibling paths, and it
is silent (multiplexing is only activated by the messaging gateway, so the
fail-closed UnscopedSecretError path never fires here).
Install set_secret_scope(build_profile_secret_scope(parent_home)) for the
build and release it in the same finally block as the home override.
Salvage-note: code relocated from tui_gateway/server.py to tui_gateway/methods_session.py (session handlers moved after the PR was opened); test patch targets unchanged — HandlerRegistry rebinds handlers onto server.py globals
Follow-up to the salvaged #59076 commit:
- Replace the bare get_secret import with a module-level Slack-pattern
helper (_get_esecret): try get_secret, on UnscopedSecretError fall back
to os.getenv. The DEFAULT profile's email adapter constructs UNSCOPED
under multiplexing, where a bare get_secret raises and would crash the
email path on startup — the exact WhatsApp defect fixed in 5438e9c629
(whatsapp_common._get_wsecret).
- Extend scope coverage to the remaining scope-blind reads:
EMAIL_IMAP_PORT / EMAIL_SMTP_PORT / EMAIL_POLL_INTERVAL (_esecret_int
replacing utils.env_int) and EMAIL_TRUST_FROM_HEADER (_esecret_bool
replacing utils.env_bool).
- Add tests: default-profile unscoped-under-multiplex construction, and
scoped ports/trust-flag no-environ-inheritance.
The email adapter (plugins/platforms/email/adapter.py) read
EMAIL_ADDRESS, EMAIL_PASSWORD, EMAIL_IMAP_HOST, EMAIL_SMTP_HOST,
EMAIL_ALLOWED_USERS, and EMAIL_ALLOW_ALL_USERS via os.getenv()
directly. In a multiplexed gateway, os.environ holds the default
profile's .env values, so every secondary profile inherited the
default profile's email credentials instead of its own.
This was a sibling of the api_server env-leak bug (#52307/#50051):
the same os.getenv→get_secret migration that PR #50094 applies to
gateway/config.py, but for the email adapter itself, which neither
PR #50094 nor #51374 covers.
Changes:
- plugins/platforms/email/adapter.py: replace os.getenv with
agent.secret_scope.get_secret for all EMAIL_* credential reads
(adapter __init__, check_email_requirements, _allowlist_in_effect,
_dispatch_message allowlist gate, _send_email SMTP helper).
- gateway/config.py: add _getenv/_getenv_str/_getenv_int helpers
(from PR #50094) and replace os.getenv with _getenv for the email
block in _apply_env_overrides, so config.platforms[EMAIL].extra
is populated from the scoped value.
- tests/gateway/test_email_secret_scope.py: 5 new tests covering
scoped credential reads, environ fallback without scope, missing-
key-no-leak, allowlist scoping, and check_email_requirements scoping.
Related: #50051, #52307, PR #50094, PR #51374
The cherry-picked #60420 hunks that converted gateway/authz_mixin.py are
dropped here: main's _auth_env/_platform_gate_env supersede them, and the
remaining authz_mixin raw-read conversions (allow-all flag + allowlists at
L459/501/879-885) land in a separate PR. Until that PR flips the allow-all
read to scope-authoritative semantics, the cross-profile environ-opt-in
inheritance case is a known gap — pin it as strict xfail so the separate PR
flips it green.
Follow-up to the review-hardening commit: the existing cases exercised
QQ_ALLOW_ALL_USERS but not the QQ_ALLOWED_USERS read at authz_mixin.py
line 459, so a revert of that line to raw os.getenv would still pass.
Add a scoped-allowlist DM case (scope admits the sender, environ does
not) plus its isolation counterpart (a secondary scope listing a
different user must not inherit the primary's environ allowlist). Both
fail if line 459 reverts to os.getenv.
Review follow-up: the adapter-level resolver alone left three paths
reading per-profile QQ_* values from raw os.getenv, so a secondary
multiplex profile's scoped opt-in or credentials were ignored (or the
primary's environ values leaked in):
- gateway/authz_mixin.py: route the per-platform allow-all flag and the
per-platform/group allowlist + allow-bots reads through the
scope-aware gateway.config._getenv. Deployment-global GATEWAY_* reads
intentionally stay on os.getenv. This makes the same fix effective
for every own-policy platform, not just QQ; unscoped behavior is
byte-identical to os.getenv.
- gateway/run.py (_own_policy_open_startup_violation): resolve the
per-platform dm/group policy and allow-all opt-in via _getenv; the
secondary-profile caller already runs inside _profile_runtime_scope.
- tools/send_message_tool.py (_send_qqbot): the QQ_APP_ID /
QQ_CLIENT_SECRET fallbacks now honor the active profile scope.
Tests: tests/gateway/test_qqbot_scope_paths.py covers all three paths
end-to-end (scope wins, no environ inheritance for non-opted profiles,
single-profile environ fallback unchanged); the STT suite now asserts
QQ_STT_BASE_URL and QQ_STT_MODEL scoping alongside the API key. All
five scoped-behavior tests fail on the previous commit and pass here.
The QQ adapter read QQ_APP_ID, QQ_CLIENT_SECRET, the QQ_STT_* backend
config and the QQ_ALLOW_ALL_USERS policy flag through raw os.getenv,
bypassing the active profile secret scope. In multiplex mode a secondary
profile whose secret lives in its own .env (installed as an isolated
scope, not into os.environ) would silently fall back to the
default/primary profile's value — the same cross-profile collision fixed
for the WeChat/weixin adapter in #59662.
Route these reads through a scope-aware resolver that reads the profile
scope when one is installed (secondary profiles and per-turn inbound) and
falls back to os.environ otherwise. The fallback is deliberate: the
primary/active profile is constructed without a scope and owns
os.environ, so a bare get_secret would raise UnscopedSecretError and
break its startup. Mirrors gateway.config._getenv.
Adds regression tests including active-profile-no-scope construction (the
fail-closed case), plus scope-wins-over-environ, two-profile isolation,
single-profile fallback, explicit-config precedence and STT key scoping.
api_server's __init__ API_SERVER_KEY read now matches the scoped
_expected_api_key path. tests/gateway/test_adapter_startup_secret_scope.py
asserts, for every migrated module: helper exists, scoped read wins, scoped
miss returns default (no environ borrow), unscoped-under-multiplex falls back
to environ without raising, and legacy single-profile reads still work.
Salvaged from #56982 (@rayjun): the live piece of the PR. The
hermes_cli/config.py get_env_value scope-honoring change and its
test_env_load_cache.py tests were already merged via ed1170cd8b
(#76462) and are dropped here.
tools/xai_http.py::get_env_value wrapped the scope-aware
hermes_cli.config.get_env_value in except Exception + a raw os.environ
fallback — swallowing UnscopedSecretError and borrowing the process
env, so a multiplexed xAI credential read could silently pick up
another profile's XAI_API_KEY. Narrow the except to ImportError (the
only legitimate degraded case) so get_secret's verdict propagates: an
unscoped multiplexed read fails closed, and a scoped miss returns the
default instead of the foreign environ value.
Co-authored-by: rayjun <rayjun0412@gmail.com>
Fixes#69379 (v2026.7.20 Docker multiplex regression: the scoped runner
reload dropped API_SERVER_* set via the container environment, silently
losing the api_server platform) by the canonical mechanism — corrects
the direction of #69524, which patched gateway/config._getenv to fall
through to os.environ on EVERY scoped miss, re-opening the
cross-profile borrow for all credentials.
API_SERVER_ENABLED / API_SERVER_HOST / API_SERVER_PORT /
API_SERVER_CORS_ORIGINS are deployment listener settings (Docker
compose environment: block, systemd Environment=), not profile
secrets: they join _GLOBAL_ENV_EXACT so get_secret reads them from
os.environ regardless of scope. API_SERVER_KEY is deliberately NOT
allowlisted — it IS a credential and stays profile-scoped, which keeps
tests/gateway/test_config.py's secondary-profile isolation semantics
intact (a secondary profile without the key still doesn't bind a
listener).
Ports #69524's regression test in the corrected form: container-env
API_SERVER_* stays visible during the scoped runner reload while the
key resolves through the profile scope; plus unit tests locking the
allowlist membership and the deliberate exclusion of API_SERVER_KEY.
The adapter's __init__ and send_weixin_direct read WEIXIN_ACCOUNT_ID/
TOKEN/BASE_URL/CDN_BASE_URL via bare get_secret, which raises
UnscopedSecretError when the DEFAULT profile's adapter constructs or
sends unscoped under multiplexing (corrects the direction of #66073 /
#68854, which tried to solve this by borrowing os.environ on every
read — a cross-profile leak).
Add a module-level _wx_secret helper following the established Slack
SLACK_APP_TOKEN pattern (#59739) and WhatsApp's _get_wsecret: a SCOPED
miss returns the default (the scope is authoritative — no environ
borrow), while an UNSCOPED read under multiplex falls back to
os.environ, which is the default profile's own value.
Regression tests cover both directions: scoped construction reads the
scope's value and a scoped miss yields empty (no borrow); unscoped
construction falls back to os.environ instead of raising.
Unquoted values truncate only at a # preceded by whitespace (so
KEY=foo#bar stays intact); quoted values scan escape-aware for the
matching close quote, keep through it, and drop a trailing '# ...'
remainder. Verified empirically against python-dotenv 1.2.2 on a
10-case corpus (full parity). Supersedes the approach in #57718, whose
scanner corrupted foo#bar-style values.
save_env_value writes values containing " or \ as escaped
double-quoted strings, and every other .env reader in the repo
(load_env/_parse_env_value, python-dotenv) reverses those escapes.
load_env_file — the parser behind build_profile_secret_scope, which
wraps every cron job and every multiplexed gateway turn — only stripped
the outer quotes, leaving the escapes literal. A credential containing
a double quote or backslash (JSON service-account blobs, generated
secrets) authenticates fine interactively but 401s under scoped
resolution, with no error pointing at the cause. Parse values with the
canonical _parse_env_value so all readers agree byte-for-byte.
test_churn_across_more_files_than_fit_in_one_argv (e65ff9625f) asserts all
1200 checked-out files read dirty before normalization. Whether git diff
content-compares an entry (seeing the CRLF churn) or trusts the stat cache
depends on racy-git detection: entries whose recorded stat is non-racy
(mtime older than the index write) read CLEAN. On CI a 1200-file checkout
straddles that boundary nondeterministically — observed 92/1200 and
661/1200 dirty on two unrelated PRs within minutes (runs 30738759530,
30738842393). Empirically reproduced: freezing a non-racy stat cache gives
0/N dirty; bumping worktree mtimes past the index write forces content
comparison and gives N/N deterministically.
Fix: bump every worktree mtime after checkout in _managed_repo so all
entries are stat-stale. Affects only the fixture; the production
_normalize_managed_eol path is untouched.
Three tests pin the exact kwargs of the picker probe call
(test_model_switch_custom_providers + two in
test_user_providers_model_switch, the latter caught by CI slice 2);
the picker-timeout change now always passes timeout explicitly (5.0 on
the non-picker path), so the pinned shapes gain the key.
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.
Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.
Based on #74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
Follow-ups on the startup-burst memo:
- Populate the memo on the valid-token fast path as well. The startup
burst usually finds a VALID token, and each check_fn call still paid
two cross-process file locks + state reads to reach that return; the
original memo only engaged after a refresh. The token has at least
refresh_skew_seconds (>=120s) of life at that return, so a 5s memo can
never serve an expired token.
- Clear the module-level memo in test_nous_portal_staging_allowlist's
refresh-capture helper: with the fast-path populate, a token memoized
by an earlier test would otherwise short-circuit the refresh these
tests assert on (3 tests failed without this).
- Add dedicated memo behavior tests (TTL hit, TTL expiry, insecure
bypass) — the original PR shipped none. Mutation-checked: all 3 fail
against main's un-memoized function, pass on this branch.
Whole-bug-class follow-up to the profiles/ gate: cron/, platforms/
pairing, and legacy pairing/ ran chown_hermes_tree unconditionally on
every boot with the identical warm-boot cost profile. Same
tree_has_non_hermes_owner gate; find evaluates the top directory first
and -quits on the first mismatch, so a mis-owned tree short-circuits in
O(1) while a clean tree pays one read-only walk instead of a full
chown -R inode rewrite.
Demote wrote the empty v23 schema via executescript inside BEGIN IMMEDIATE,
which commits early and can leave trash + empty indexes without rebuild
markers. Re-run then tore down trash and stamped fts_storage_version with
docsize=0, permanently losing historical session search.
Stage markers with the demote, create schema only after they are durable,
heal empty-index bookkeeping on resume, and refuse settle until the base
index is populated. Settle refusal returns ok=False instead of raising,
and resume fails fast if the base v23 table cannot be re-created.
Orphan-marker repair only resets a missing fts_rebuild_progress to 0 once
the index is known empty: the chunk worker replays its whole selected id
range without an anti-join, so a partially indexed DB that lost only its
progress key is first reset to a known-empty surface, then rebuilt.
Ported onto the SessionDB mixin split (hermes_state_search.py /
hermes_state_schema.py).
The RO branch's new FTS capability probe raises sqlite3.DatabaseError on
a malformed store (the probe itself only catches OperationalError). The
outer __init__ handler re-raises without closing self._conn, leaking a
tracked connection for the process lifetime — which makes
_backup_db_file refuse its raw-copy, so the writable heal that follows
(web_server's stale-schema/malformed reopen) repairs the store WITHOUT
the forensic backup repair_state_db_schema promises. Close-then-reraise
on any probe failure, mirroring _open_probed's cleanup discipline.
Regression test: corrupt sqlite_master (duplicate messages_fts row),
assert the failed RO open leaves no live tracked connection and the
subsequent writable heal creates its malformed-backup file. Mutation-
checked: no-oping the cleanup handler makes the test fail.
The terminal-command guard path loaded config 2-3x per invocation via
load_config(), which pays a defensive deepcopy of the entire config on
every call (~356us of the ~376us warm-cache cost measured on a real
config.yaml). All six swapped call sites were audited read-only — every
caller takes scalar reads or iterates the returned structures; none
mutate (the save path at save_permanent_allowlist keeps load_config) —
so they now use load_config_readonly(), the API built for exactly this
(precedent: #74211, #74322; the one unsafe-site lesson from #56085's
salvage is covered by the mutation audit and a cache-integrity test).
Measured (real config.yaml, warm cache): load_config 376.0us ->
load_config_readonly 19.9us (18.9x); full guard pass
check_all_command_guards('ls -la','local') 930.7us -> 241.8us (3.85x).
Tests: new test_approval_config_readonly.py drives the real functions
against a temp HERMES_HOME — readonly call counts per function, a
no-deepcopy pin for the full guard pass, and cache-identity/integrity
checks. Existing test mocks retargeted from load_config to
load_config_readonly (same injection intent). Note: 6
test_approval_mode_parity failures are pre-existing ordering flakes —
identical with the change stashed on clean main.
Review follow-up on the #76098 salvage: the 4096-entry count bound alone
doesn't bound MEMORY — write_file/patch argument strings run 100KB+, so
a long-lived gateway process under sustained heavy write workloads could
pin ~800MB of evicted-session strings. A 32MB byte budget extends the
existing FIFO eviction; common-case args (0.5-2KB) never hit it. New
guard test mutation-checked (fails with the byte leg disabled).
The pre-send normalization pass re-canonicalized every historical tool
call's argument JSON on every API-call iteration — quadratic in session
tool-call count. Route it through a bounded value-keyed memo (the
_MSG_TOKENS_CACHE idiom from agent/model_metadata.py): per-iteration
cost is now proportional to new tool calls, not all of them.
Measured (simulated growing session, repo venv): session-total
canonicalization cost 1056 ms -> 58 ms at 500 tool calls x 2 KB args
(18.3x), 5744 ms -> 79 ms at 500 x 16 KB (72.5x). Byte-parity with the
pre-fix logic is asserted at every iteration (unicode, nested,
malformed, empty, non-string args), and a call-count test proves
json.loads invocations went from K(K+1)/2 to K per session.
Review follow-up on the #76142 salvage: MemoryStore path-resolves and
shares one process-wide connection per file, so MemoryStore(":memory:")
creates a literal ./:memory: FILE whose state leaks across test runs —
the second run of the file failed all three spy tests because the
NULL-vector test had permanently wiped hrr_vector in the leaked db.
tmp_path isolates each run; verified two consecutive runs green + full
tests/plugins/memory/ green.
FactRetriever.search() re-encoded the query vector once per candidate,
related() re-encoded both role atoms once per fact row, and probe()
re-encoded the role-content atom once per row. All three encoders are
deterministic (SHA-256 counter blocks), so the hoisted vectors are
bit-identical to the per-iteration values they replace.
Measured (300-fact store, dim=1024, median of 30 calls): search()
11.62 -> 1.46 ms/call (8.0x; encode_text 30 -> 1 per call), related()
63.08 -> 16.17 ms/call (3.9x; encode_atom 601 -> 3 per call), probe()
431.93 -> 389.36 ms/call (1.1x; dominated by per-fact content encoding,
which is inherent to the algorithm and unchanged).
Tests: call-count regression tests for each hoist plus a bit-exact
parity test of search() against the pre-fix per-candidate loop.
Every ORDER BY id query on the messages table sorted or scanned the
whole session: get_messages_around's window seek, latest_message_row_id
(LIMIT 1), and get_messages' full-load ordering all paid O(session
history) per call — hot mid-turn via session_search and reactions.
messages.id is an original column (INTEGER PRIMARY KEY AUTOINCREMENT),
so the index lives in SCHEMA_SQL next to idx_messages_session — no
legacy-column migration hazard (the kanban lesson from #28776 does not
apply).
Measured (real schema, one 20k-message session, median of 30):
get_messages_around 7.08 -> 0.22 ms (32x), latest_message_row_id
3.37 -> 0.011 ms (307x), get_messages full load 111.6 -> 98.6 ms
(1.13x — remaining cost is row deserialization, not the sort).
Window results byte-identical at probe points across the session.
Tests: VM-step pin (get_messages_around bounded work, calibrated
~12 vs ~855 handler calls, threshold 300 — fails without the index)
and window parity with/without the index. No EXPLAIN/plan text
(behavior contracts, AGENTS.md).
The scheduler's pre-dispatch loop called advance_next_run per due job —
one full load_jobs() + one full save_jobs() of the jobs file each — so
N due jobs cost N reads + N writes of the whole file (gateway-restart
catch-up or co-scheduled bursts). advance_next_runs() does one load +
at most one save for the whole due set with identical per-job semantics;
advance_next_run() is now a thin wrapper over it.
Measured (50 due recurring jobs, real jobs file): 107.9 ms -> 2.5 ms
(45x; 50 loads + 50 saves -> 1 + 1).
Tests: batch advances recurring and skips one-shots, single load + save
I/O pin (fails pre-fix — no such function), no save when nothing
advances, and per-job wrapper semantics unchanged. Related: #60946 and
#75833 both restructure this loop's call site for correctness — neither
addresses the I/O cost, and this batch primitive composes with either
dispatch design; happy to rebase onto whichever lands first.
Every reply paid one extra Discord API round trip: the text send path,
the voice send path, and the edit path each called fetch_message() just
to obtain a reference or an editable handle. Discord resolves
message_reference payloads from ids alone, and PartialMessage.edit()
works without a fetch — so build MessageReference directly (with
fail_if_not_exists=False, preserving the deleted-target behavior the
existing send-side 10008 retry already covered) and use
channel.get_partial_message() for edits. Overflow continuations keep
threading via an ids-built reference fallback for PartialMessage.
Measured by call count (deterministic): reply sends and edits now make
ZERO fetch_message calls where they made 1 per reply and 1 per edit
(including every streaming edit tick).
Tests: pin that first-mode replies construct the reference without any
fetch, deleted-target retry test updated to assert fetch await_count==0
(retry now happens purely send-side), overflow/edit mocks retargeted
from fetch_message to get_partial_message (any fetch regression breaks
all five). Note: 4 discord-suite failures are pre-existing ordering
flakes — identical with the change stashed on clean main.
Architecture fix for the bug class behind the Termux --version NameError
(live on main since eb4040242): version-printing kept being reimplemented
as *_fast() copies at the top of hermes_cli/main.py, each duplicating
canonical logic (project-root resolution, container detection, profile
detection). The copies drift silently — eb4040242 edited the canonical
output and referenced the PROJECT_ROOT module constant inside the fast
function, which doesn't exist yet at the fast exit point.
- hermes_cli/_startup_fast.py: THE implementations, stdlib-only. main.py's
*_fast() names become thin delegates (kept for test/back-compat), and
PROJECT_ROOT itself derives from the same helper — the constant and the
fast path can no longer disagree.
- Fast output now includes the .install_method stamp (one cheap file read)
and a 'Run hermes version for update status' pointer, so globalizing the
fast path doesn't silently drop slow-path info.
- Guard tests: (1) import-weight — subprocess-imports _startup_fast and
fails if any heavy module (config/yaml/argparse/cli/run_agent/httpx)
lands in sys.modules; (2) subprocess parity on+off Termux — the test
that would have caught eb4040242 the day it landed; (3) install-method
stamp surfacing.
hermes --version: ~3.8s cold / 0.2-0.4s warm -> 0.01-0.02s everywhere.
- agent_import.dump_yaml_file now calls utils.atomic_yaml_write instead
of hand-rolling safe_dump + atomic_write_text — same temp+fsync+atomic
rename and symlink preservation, plus mode/owner preservation a
0600-secured config.yaml needs
- openclaw script: the EXDEV/EBUSY copy fallback gains copystat + target
fsync so the docstring's 'mirrors utils.atomic_replace' durability
claim is true on cross-device deployments
- trim load_yaml_file's docstring to the behavior contract
The inlined temp-file + os.replace in openclaw_to_hermes.dump_yaml_file
replaced a symlinked config.yaml with a regular file, silently detaching
managed deployments that symlink ~/.hermes/config.yaml into a dotfiles repo or
profile package. The bare path.write_text it replaced followed the link, and
utils.atomic_replace -- which the hermes_cli twin reaches through
atomic_write_text -- resolves the link for exactly this reason (#16743).
Mirror that here: resolve the symlink before creating the temp file so the
rename lands on the real file, and fall back to copyfile on EXDEV/EBUSY now
that the target can live on another device. Covered by a regression test that
fails when the resolution is removed.
Also guard the permission-denied test for Windows: os.geteuid does not exist
there and chmod-based denial is unreliable, so skip on non-POSIX.
agent_import.py carries a private load_yaml_file/dump_yaml_file pair that
returned {} for an absent file AND for a present file it could not read or
parse. Three importers -- import_permission_allowlist, import_permission_denylist
and import_mcp_servers -- read config.yaml through it, merge one section into
the result, and write the whole mapping straight back. So a YAML syntax error,
a permission problem or a broken mount meant the importer replaced every
setting the user had with only the one to three keys it merged, and still
reported the item as "imported". The write was a bare path.write_text(), so an
interrupted import truncated the file instead.
Distinguish the two cases at the read. Absent, or present but empty, still
yields {} so first-time creation works. Present but unreadable, unparseable, or
not a mapping raises ConfigReadError; the three sites funnel through a new
load_target_config() that records the refusal as a per-item error and leaves the
file byte-identical. Dry-run refuses too, rather than previewing an "imported"
that would destroy the config. dump_yaml_file now writes through
utils.atomic_write_text, which the module already imports and uses for the
memory store.
This is the invariant hermes_cli/config.py already enforces for its own writers
via require_readable_config_before_write / atomic_config_write, whose docstring
names this exact root cause and calls itself "the single chokepoint every
config-update path should use". agent_import.py has its own helper pair and so
was never covered; it was the last config.yaml writer without the guard.
The identical helper pair lives in openclaw_to_hermes.py, the script this module
was ported from, where twelve config.yaml read-modify-write sites share the same
defect; fixed there too. Its refusal is recorded at the run_if_selected dispatch
point, which flips the existing _config_apply_blocked flag so the remaining
config-mutating options short-circuit instead of each rediscovering the same
unreadable file. The atomic write is inlined with tempfile + os.replace because
that script runs standalone with only the stdlib on its path.
kshitij