The bounded flush-scan in _flush_messages_to_session_db_unlocked skips
the identity-matched prefix of its previous snapshot, on the documented
assumption that no code path pops _DB_PERSISTED_MARKER from a live dict
in place. finalize_turn's pure-tool-call-tail fill is exactly that path:
it pops the marker so the filled content gets re-persisted — but the
cursor then skips the row anyway, so the delivered final response never
reaches state.db and /resume replays content="" (the #43849/#44100
class resurfacing via the perf cursor). Invalidate the cursor at the
pop site so the filled row is re-examined.
When model is a bare scalar (e.g. 'model: gpt-4o'), running
'hermes config set model.provider openai' silently destroyed the
model id because _set_nested replaced the scalar with an empty dict
before writing the sub-key. Now the scalar is normalized to
{default: <id>} first, preserving the model id.
Companion to the preflight fix: _estimate_msg_budget_tokens charged
reasoning_details at chars/4 via _REPLAY_BUDGET_KEYS, so the signed/base64
envelope (measured 72% of the reasoning mass on Anthropic-wire sessions)
consumed the tail budget and _find_tail_cut_by_tokens summarized away real
transcript to make room for tokens that are never sent (69 messages on the
measured session; up to ~4.8x budget inflation on thinking-heavy histories).
Per the #51800 counter-argument, actual thinking TEXT stays visible to the
budget: _reasoning_details_text_chars counts thinking/text/summary fields
and skips signature/data/encrypted blobs, and the text is skipped entirely
when reasoning/reasoning_content already carries the identical prose (so it
is charged once, not twice). codex_reasoning_items remains fully charged —
Codex Responses genuinely replays it every request (#55572).
Sabotage-verified: restoring reasoning_details to _REPLAY_BUDGET_KEYS fails
the new envelope and double-charge tests.
The reasoning_details field (OpenRouter/Anthropic thinking blocks +
opaque cryptographic signature blobs) inflates the rough token estimate
by ~4x. Providers do not bill these envelope bytes as prompt tokens.
In a measured Kimi K3 session, reasoning_details held 2,124K chars
vs 281K chars of actual thinking text. The estimator reported ~533K
tokens when real prompt_tokens was ~140K — triggering compression at
~27% of the configured threshold.
Fix: skip reasoning_details in both _estimate_message_chars and
_estimate_message_tokens_without_images, alongside the existing
_anthropic_content_blocks exclusion.
Fixes#73298
Fix-up for the cherry-picked cooldown persistence: the PR's tests mocked
the DB (SimpleNamespace(_db=MagicMock())), which cannot prove the cooldown
survives a restart. Replace with the production shape — a real SessionDB
on disk behind the real AsyncSessionDB facade — and add a restart
regression: fail a hygiene compression on runner #1, tear it down, build a
fresh GatewayRunner on the SAME database, and assert the cooldown is still
honored (no compression agent instantiated). Also updates the timeout test
to assert the DB-backed record_compression_failure_cooldown write instead
of the removed in-memory dict.
Sabotage-verified: reverting gateway/run.py to the in-memory dict makes
the restart test fail.
The session hygiene compression path tracked its per-session failure
cooldown in an in-memory dict (_hygiene_compression_failure_cooldowns).
When the gateway restarted, the dict was gone, so the next message
re-triggered the same failing compression, wedging session storage.
The state DB already has a persistent column
(sessions.compression_failure_cooldown_until) and full read/write/clear
methods (record_/get_/clear_compression_failure_cooldown in hermes_state.py)
used by the in-conversation compression path (context_compressor.py) but
not by the session hygiene path in gateway/run.py.
Fix: replace the in-memory dict with calls to the persistent DB methods:
- Cooldown check: use get_compression_failure_cooldown instead of dict lookup
- Timeout failure: use record_compression_failure_cooldown instead of dict write
- Abort failure: use record_compression_failure_cooldown instead of dict write
After a restart, a session whose compression is in cooldown is now skipped
for the cooldown's remaining duration rather than re-attempted immediately.
Fixes#74136
Follow-up to the boundary clamps: drive the REAL compress() pipeline over
the live 8-message transcript shape from #75588 (system + tool-only
suffix, aligned head == len(messages)). Asserts no exception, the summary
LLM is never invoked when the window is out of range, and the transcript
is returned unchanged. Sanity-checks the clamped tail-cut boundary so a
future regression of the n+1 floor is caught even where downstream
callsite clamps mask the IndexError.
Fix#75588
## Root cause
When a short conversation ends in a tool-call/result group and the
protected head alignment reaches the end of the message list,
_find_tail_cut_by_tokens() could return len(messages) + 1. This
happened because the final return used max(cut_idx, head_end + 1)
which could push past the array length when head_end >= len(messages).
The out-of-range value then propagated into _find_context_summaries()
which iterated range(start, end) and indexed messages[idx] without
clamping, raising IndexError and failing the active gateway turn.
## Fix
Two-layer defense:
1. Source fix: _find_tail_cut_by_tokens() now clamps its return to
min(n, ...) so it never exceeds len(messages).
2. Defensive clamp: _find_context_summaries() now bounds start/end
to [0, len(messages)] so even if a future caller passes bad values,
it cannot crash.
## Verification
- 7 new regression tests for the exact boundary conditions
- All 214 existing test_context_compressor.py tests pass
2de1e86c16 appended updated versions of five doctor tests without
removing the originals; the earlier definitions were silently shadowed
(dead) and tests/test_no_shadowed_test_definitions.py now fails on every
PR slice that runs it. Keep the later (runtime-winning) definitions,
delete the stale earlier ones.
Three fixes for concurrent MCP server spawn races in register_mcp_servers()
and discover_mcp_tools():
1. register_mcp_servers: add k not in _server_connecting guard to the
new_servers filter. Without this, a concurrent second call sees the
same servers as 'new' and spawns duplicate stdio subprocesses.
2. discover_mcp_tools: same _server_connecting guard in the
new_server_names filter. This entry point is called from CLI, TUI,
gateway, and cron — any two racing would double-spawn.
3. Stale _server_connecting cleanup on TimeoutError/InterruptedError.
When _run_on_mcp_loop times out or is interrupted, _discover_all's
gather may not have finished, leaving entries stranded in
_server_connecting that block future reconnection attempts. The
cleanup clears only entries added by this call (not external ones),
logs a warning, and records connect errors.
Salvage of #58879 by @nanami7777777 (superset of #58867 by @liuhao1024).
Adapted to current main which has evolved significantly since July 5.
Closes#58862Closes#58867Closes#58879
Twenty-odd desktop files import `@assistant-ui/core` directly, but it was
never in `dependencies` — it resolved only because react 0.15 pulled core
0.3.2 and npm hoisted it to the workspace root. react 0.14.24 wants core
^0.2.19, which nests under `react/node_modules`, so the undeclared imports
stop resolving and the build fails on `@assistant-ui/core/internal`.
Declare it so resolution doesn't depend on hoisting luck either way.
The 0.15 migration swapped the `useMessageRuntime()` hook for a plain
`useAui().message` accessor read. The hook subscribes and re-renders on
change; the accessor does not, so message components paint once and then
go stale until the window is reloaded. `useComposerRuntime().subscribe`
was widened to `aui.subscribe` the same way.
Restores the 0.14 call form, including the test mock whose added
`getState` stub let the API break through CI unnoticed.
The 0.15 bump rode along with the npm audit pass in #75037, but it was
not required by any advisory — `npm audit` reports 0 vulnerabilities with
0.14.24 pinned. It is a breaking major that costs us a working transcript,
so take the API stability instead.
Follows PR #75592: the notifier is no longer gated on
kanban.dispatch_in_gateway. Every gateway delivers events for
subscriptions owned by the profiles whose adapters it hosts; legacy
unstamped subscriptions go only through the confirmed dispatcher
lock owner. Adds a 'Multi-profile setups' subsection to the kanban
Gateway notifications docs (en + zh-Hans).
Review follow-up on #75102. The shadow substituted the sidecar whenever
the ``api_content`` key was merely PRESENT, but the wire only substitutes
a non-empty string sidecar on a user/assistant row (see
``turn_context.substitute_api_content``). For any other shape the sidecar
is popped and discarded while the clean ``content`` is sent -- so the
shadow dropped real content from the estimate and UNDERcounted, the
dangerous direction: compaction fires too late and the turn dies on a
hard context-length error instead of merely compressing early.
Gate the substitution on the same predicate, and cover the divergent
shapes (None, empty string, int, list, non-user/assistant role) with a
test that fails against the unconditional version.
Also rename the image test: it never carried a sidecar, so it was not
testing what its name claimed. It is a non-regression pin on the flat
per-image accounting that moved into ``_wire_message_shadow()``, and is
now named for that.
`api_content` is a SUBSTITUTE for `content`, not an addition to it.
`turn_context.substitute_api_content()` pops the sidecar and overwrites
`content` at every API-bound message-build site (the `api_messages` build
in `conversation_loop`, the max-iterations summary in
`chat_completion_helpers`, the chat-completions transport), so exactly one
of the two is ever sent to the provider.
The preflight estimator counted both, because both `_estimate_message_chars`
and `_estimate_message_tokens_without_images` walked every key of the
persisted dict with a single-entry denylist (`_anthropic_content_blocks`).
Any message whose sidecar differs from its clean stored content was counted
twice — exactly 2.00x on a 40KB sidecar.
The sidecar exists to keep the provider prompt-cache prefix byte-stable, so
it is written on precisely the long, cache-pinned messages where the
doubling hurts most. Because `estimate_messages_tokens_rough()` also feeds
the compaction threshold via `context_compressor` and `conversation_loop`,
the inflated estimate makes compression fire on phantom bytes.
Fix: substitute rather than sum, mirroring the wire. The two estimator
helpers had drifted into near-identical copies of the same shadow-building
loop, so this factors the shared logic into `_wire_message_shadow()` and
fixes the class once instead of patching one site and leaving the other.
Image accounting is unchanged: base64 payloads are still replaced with a
placeholder and charged at the flat `_count_image_tokens` rate, and the
`_multimodal` text_summary path is preserved.
Tests: three cases in `TestEstimateMessagesTokensRough` — sidecar equal to
content is counted once, a sidecar that DIFFERS is still counted (a lower
bound, so it fails if the field were dropped rather than substituted, which
would undercount the real request), and a sidecar cannot smuggle raw base64
past the flat image rate.
Verified on Linux (Python 3.11): 53 passed in
tests/agent/test_model_metadata.py, 57 passed with
tests/agent/test_context_breakdown.py, 656 passed / 3 skipped across the
compression/context/token/estimate/prune surface of tests/agent.
Mutation-tested: reverting the substitution fails the new equality test.
`scripts/check-windows-footguns.py` is not applicable — no file I/O,
process management, terminal handling, subprocesses, or signals.
The embedded Hindsight daemon's profile env file carries the plaintext
HINDSIGHT_API_LLM_API_KEY but was written via bare write_text(), leaving
it with umask-derived (typically world-readable) permissions.
- Create/truncate the file via os.open(..., 0o600); chmod a pre-existing
file to 0600 BEFORE writing new secret bytes.
- Post-write validation on POSIX: verify 0600, retry chmod, and raise if
the file still isn't owner-only.
- If validation fails, unlink the secret file so a plaintext key is never
left behind with unverified permissions.
- Regression tests under tests/plugins/ for fresh-write mode, tightening a
pre-existing 0644 file, and cleanup on validation failure.
Narrowed reimplementation of #74236 confined to plugins/memory/hindsight/;
the core utils.py atomic-replace opt-out from the PR was dropped.
Co-authored-by: carrion256 <carrion256@proton.me>
Follow-ups to the previous commit (#74414 by @webtecnica, re #74373):
- When distribution_owned is OMITTED, restore the legacy contract: every
staged entry outside USER_OWNED_EXCLUDE is copied. The cherry-picked
filter consulted owned_paths(), which silently narrowed omitted-list
distributions to DEFAULT_DIST_OWNED and dropped undeclared payload
(extra top-level files/dirs existing distributions legitimately ship).
- Make explicit allowlists path-aware so documented nested entries like
skills/research/ and cron/digest.json select exactly that subtree/file
instead of being dropped by the top-level name comparison. Traversal
segments (.., absolute) and USER_OWNED_EXCLUDE roots are still rejected.
- Regression tests: omitted-list legacy behavior + nested-path allowlist.
_copy_dist_payload() in profile_distribution.py iterated all staged
entries without consulting the manifest's distribution_owned allowlist,
so manifests that restricted distribution_owned only had cosmetic effect.
Fix: compute manifest.owned_paths() at the top of _copy_dist_payload()
and skip entries not in that set, after the USER_OWNED_EXCLUDE check.
The owned_paths() method already existed on DistributionManifest and
correctly falls back to DEFAULT_DIST_OWNED when no explicit
distribution_owned is set, so the new filter preserves backward
compatibility for existing manifests.
Closes#74373
Follow-ups to the previous commit (#74155 by @Drexuxux):
- enrich_model_switch_warnings_for_gateway() -> merge_preflight_compression_warning()
still called the sync resolve_display_context_length() provider probe ladder
inline in both async /model call sites; dispatch it via asyncio.to_thread.
- Replace the inspect.getsource() test (source-reading tests are banned by
AGENTS.md) with behavioral tests that drive the real _handle_model_command:
assert the resolver runs off the loop thread and that the warning enrichment
is dispatched through asyncio.to_thread.
resolve_display_context_length() runs two blocking chains: the route
comparison in should_clear_context_pin() and the provider probe ladder in
get_model_context_length() (blocking requests calls to Anthropic /v1/models,
Copilot, Nous, Codex, GMI, Ollama, models.dev and OpenRouter).
The gateway message path already offloads both via
get_model_context_length_async() and should_clear_context_pin_async(), but
the /model slash-command handlers (_handle_model_command, _finish_switch)
called the sync helper directly, freezing the whole event loop for the
duration of the probe ladder - no messages processed on any platform, and
the Discord heartbeat timeouts that get_model_context_length_async() was
introduced to prevent.
Add resolve_display_context_length_async(), a thin asyncio.to_thread wrapper
mirroring the two existing *_async helpers (no logic duplication), and await
it at both handlers.
setup_path() only wrote a 'hermes' launcher into the command-link dir,
even though pyproject.toml declares three [project.scripts]:
hermes, hermes-agent (run_agent:main), hermes-acp (acp_adapter.entry:main).
Fresh venv installs (the common case on macOS/Linux) leave
~/.local/bin/{hermes-agent,hermes-acp} empty, so external tools
expecting 'hermes-acp' as a standalone command (documented as
first-tier supported in website/docs/user-guide/features/acp.md)
fail to find it after a fully successful install.
Loop over the three console-script names, writing a shim per entry
that exec's the venv interpreter with the right checked-in
entrypoint. --no-venv keeps the old single-shim behaviour since
it does not manage the venv and only 'hermes' is guaranteed on PATH.
Fixes#74819
_is_env_config_key() already routes _API_KEY and _TOKEN suffixed
keys to .env for safe credential storage. Add _SECRET to the suffix
list so keys like CLIENT_SECRET and ENCRYPTION_SECRET are stored
in .env (excluded from git by default) rather than config.yaml.
7b5a18817 migrated the sibling slug sites to custom_provider_slug, which
keeps a keyed providers: entry's config key as its durable identity. It
covered find_custom_provider_identity_by_model; canonical_custom_identity's
third recovery source - the configured-provider fallback - still built
f"custom:{normalized}" out of whatever string the caller happened to hold.
_get_named_custom_provider matches on either spelling, so a display name
that differs from its config key matches the entry and then heals to
custom:<display-name>. That is a second identity for one endpoint: the
endpoint- and model-based sources of the same function return
custom:<config-key>, and so does everything that persists or restores a
session's provider override. canonical_custom_identity exists precisely to
make a bare "custom" routable again, and tui_gateway calls it on the
session-persist, resume and recovery paths - so the divergence lands in
stored session identity.
Re-resolve through the endpoint the matched entry owns, reusing the
function's own URL-based canonicaliser rather than duplicating the match
logic. Legacy unkeyed custom_providers: entries keep their name identity,
and an unconfigured candidate still returns None.
524ab5399 widened the media read timeout from send_video to "all upload send
paths" - send_voice/send_audio/send_photo/send_document/send_media_group/
send_animation. Both send_photo calls inside send_image() were missed, so
they still ran on the short timeout the rest of the Bot API is tuned for
while the sibling media paths already pass it.
The missed pair is the worst one to miss: send_image tries a URL send first,
then falls back to downloading the image and uploading the bytes - the path
documented as "supports up to 10MB", i.e. the slowest send in the file and
the one whose server-side processing wait most often outlasts the short
budget. When it times out the handler's last resort posts the bare URL as
text, so the picture silently never arrives as a picture.
Pass _MEDIA_SEND_READ_TIMEOUT on both, covered by two behavioral tests that
drive send_image for real - the URL send and the forced byte-upload fallback
- and assert the read_timeout that actually reaches the Bot API.
Address review feedback from teknium1:
1. Re-validate stale 'test' category entries in quick() — existing
tracked.json entries under now-protected directories (patches/,
projects/, etc.) are re-classified via guess_category() and
dropped instead of deleted, mirroring the cron-output pattern.
2. Add patches, projects, skins, themes, contributors to
_EMPTY_DIR_PROTECTED_TOP_LEVEL so the empty-directory sweep
never traverses into these user-authored project trees.
guess_category() classified any file whose name starts with 'test_' or
'tmp_' as disposable, even when the file lived under user-authored
directories like patches/, projects/, skins/, or themes/. Files in
these trees were silently deleted on session end.
Added the missing user-project directories to the exclusion list so
that basename-based classification only applies to files in temporary
or scratch locations, not durable project trees.
Two call sites in _busy_steer_command were assigning directly to
adapter._pending_messages[quick_key], which overwrites the FIFO head
when a message is already enqueued. Changed both to self._enqueue_fifo()
which preserves the pending slot and appends to the overflow tail.
Regression test verifies both the pending-sentinel and no-steer() paths.
Ports the negative limit/offset fix onto the current router modules
(hermes_cli/web_routers/sessions.py, profiles.py) since the handlers
moved out of web_server.py in 011ec4513e after this PR was opened.
Per review feedback: only add Query(..., ge=0) — no le=500. The
messages route already clamps oversized requests with min(limit, 500)
and must keep that behavior (succeed + cap) rather than reject them;
the two session-list routes never had a public 500 cap and shouldn't
gain a new rejecting one as a side effect of this fix.
_resolve_explicit_runtime's generic-provider branch accepted model_cfg's
persisted api_mode unconditionally, letting a stale mode from a previous
provider (e.g. anthropic_messages) leak into a newly-switched provider
(e.g. gemini) and break the transport. Reuse the existing
_provider_supports_explicit_api_mode guard, already used by the copilot
and named-custom-provider resolution paths for exactly this case, so the
persisted mode is only honored when model_cfg's provider matches the one
being resolved.
Closes#74318
agent_init.py's init-time fallback and chat_completion_helpers.py's
try_activate_fallback() still read key_env via raw os.getenv(), missing the
per-profile secret scope installed by the multiplexed gateway (same bug
fixed for fallback_config.py/auxiliary_client.py in this PR). Both now
delegate to hermes_cli.fallback_config.resolve_entry_api_key(), and the
Ollama Cloud OLLAMA_API_KEY read now goes through
agent.secret_scope.get_secret() too.
agent_init.py's fallback loop had no try/except around key resolution
(unlike the other three call sites), so a fail-closed UnscopedSecretError
under multiplexing would have crashed init instead of skipping to the next
fallback entry — added the same skip-and-continue handling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
resolve_entry_api_key() and the duplicated _fallback_entry_api_key()
read key_env via a raw os.getenv(), bypassing per-profile secret
scoping in the multiplexed gateway. Under multiplexing this can hand
a fallback request another profile's credential. Both now resolve
through agent.secret_scope.get_secret(), which reads the active
profile scope when multiplexing is on and falls back to os.environ
unchanged when it's off, so single-profile behavior is preserved.
Closes#74311
ForwardedRecordsParseMiddleware.handle() called the coroutine function
_send_loading_heartbeat() without awaiting it, so the coroutine was built
and dropped. The RUNNING heartbeat never reached the client and Python
raised "coroutine was never awaited".
Forwarded-record parsing is the slow inbound path, which is where the
loading bubble matters most: the user sees nothing while the deep parse
runs. Awaiting is safe, since the helper already swallows every exception
and the call sits inside the middleware's own try block.
Python keeps only the last definition of a name in a scope, so a duplicate
silently deletes the first. Four had accumulated, and two cost real coverage.
tests/agent/test_auxiliary_client.py grew a second _clean_env autouse fixture
alongside an NVIDIA feature. Being module-level with the same name, it
replaced the original for all 158 tests in the file: the ANTHROPIC_API_KEY,
ANTHROPIC_TOKEN, CLAUDE_CODE_OAUTH_TOKEN, OPENAI_MODEL, LLM_MODEL and
NOUS_INFERENCE_BASE_URL stripping went away, and so did the _aux_unhealthy_*
cache reset between tests. Two tests had already started clearing that cache
by hand to work around the leak, one of them with a comment describing the
pollution. The NVIDIA keys are folded into the original fixture and the
duplicate removed.
tests/gateway/test_mattermost.py had two copies of
test_progress_send_with_invalid_thread_root_never_falls_back_flat. The
surviving copy omitted the recorded 400 "invalid root_id" state, so the case
the name describes never ran. Both now run under distinct names.
The other two were harmless but hid the pattern: an exact duplicate in
test_tts_media_routing.py, and a dead _codex_auth_store in
test_credential_pool.py that nothing calls.
tests/test_no_shadowed_test_definitions.py walks every test module and fails
on any repeated definition in one scope, exempting the property/setter/
register family and throwaway _ callbacks. It fails on main listing exactly
these four.
/background (/bg, /btw) exists to start independent work while the current
turn keeps running. Typed while the agent was busy it went into
_pending_input like ordinary input, and process_loop is blocked inside
self.chat() for the whole run, so the background task only started once the
foreground turn had finished. That is the one moment it was not needed.
/steer had the identical problem and was fixed the same way, by dispatching
inline on the UI thread. The command's own CommandDef already declares
busy_policy="dispatch"; the gateway honours that, the classic CLI never
consulted it.
The foreground turn is untouched: no interrupt, no steer, and ordinary
non-slash input keeps following the configured busy-input behaviour.
append_message refused immediately when another writer held the session's
compression lock. The conversation loop turns that into
session_persistence_failed and tells the operator to check disk space and
permissions, when the store is healthy and merely busy. The two append
attempts in the reported incident were 3ms apart, so there was no wait at
all before the turn was destroyed (#75083).
The wait is deliberately short (_COMPRESSION_BUSY_WAIT_S, 5s) rather than
the 60s transcript write patience. The lease is a correctness boundary, not
just a busy signal: test_compression_lease_blocks_non_owner_but_allows_owner_flush
pins that a late stale turn must not land in a session being compressed.
Reusing the full write patience made that append succeed once the lease
aged out, which is exactly what the guard exists to prevent. A short budget
saves the common case, where compression publishes in a couple of seconds,
and still refuses a writer locked out by a long-running or wedged
compression.
CompressionSessionBusyError could not simply be retried either: it covers
two conditions with opposite handling. A compressor discovering its own
lease is gone is permanent, and retrying that would spend the whole budget
before failing anyway. Split the transient case into a
SessionCompressionInProgressError subclass, raised only by append_message,
and wait on just that. Existing except CompressionSessionBusyError handlers
catch both unchanged.
The retry jitter is extracted into _sleep_before_write_retry so the lock
path and the compression path share one implementation.
Answering "Always Approve" on the /clear, /new, /reset and /undo
confirmation calls save_config_value("approvals.destructive_slash_confirm",
False) and then appended "Future /clear, /new, /reset, and /undo will run
without confirmation" unconditionally.
save_config_value catches its own exceptions and reports the outcome in the
return value, so the caller's try/except could never observe a failed write,
and the return value was ignored. On any install whose config.yaml is not
writable the user was told the preference stuck when it had not, and the
prompt returned on the next restart with no explanation.
Check the return value. The approved action still runs either way, but when
the write did not land, say so and point at the config key instead of
promising an opt-out that was never written.
shutil.move() moves into an existing destination directory instead of
replacing it. When the snapshot extract failed after creating some of its
output, the recovery path moved each staged entry onto a path the extract
had already created, burying the user's own skill one level deeper
(skills/alpha/alpha/) and leaving the snapshot's partial content in its
place. rollback() then returned "snapshot extract failed (state restored)".
Clear the failed extract's output before moving the staged copies back:
entries the original tree never had are dropped, and each staged entry's
destination is removed before the move. When an entry still cannot be
restored, keep the staging dir and name the entries in the message rather
than reporting a restore that did not happen.
spfcraze