SessionDB(read_only=True) cannot create a missing store, so a fresh install
running hermes insights / /insights errored instead of reporting no data
(reported by @ehz0ah on #110718; guard shape from @kshitijk4poor's #110026).
Co-authored-by: kshitijk4poor <kshitijk4poor@users.noreply.github.com>
Two write paths were still reachable through the now read-only foreign-profile
handle (reported by @ehz0ah on #110718): the repo-root backfill in
_discover_repos_payload raised and was swallowed per RPC, silently dropping the
persistence; the Bot Chat unarchive in session.list's exact-title lookup failed
the RPC with error 5006. Backfill now skips on read-only handles (that
profile's own gateway backfills on its refreshes); the unarchive escalates to a
short-lived registry writer for the rare recoverable-archive case.
Reported-by: ehz0ah
_profile_db acquired a registry WRITER on another profile's state.db for
every RPC about it and closed it in the handler's finally — schema init
plus the write lock on a store owned by that profile's own gateway, once
per sidebar refresh under desktop app-global remote mode. Reads now go
through the dashboard sidebar's read-only open helper; the four handlers
that mutate (session.move_cwd, session.delete, session.set_hidden,
session.foreign.import) opt in with writer=True.
Field trace and read-only approach by @samalone on #109737.
Co-authored-by: samalone <samalone@users.noreply.github.com>
hermes insights and /insights only read; taking the default writer runs
schema init and contends the write lock on a live store.
Salvaged from #109737.
Two SessionDB handles on one state.db in one process see the same
descriptors; the first release must not drop the range the second still
needs. Test covers the same-process sibling and the foreign-process close.
iter_deleted_sqlite_sidecar_holders() returned [] on every non-Linux
platform, so refuse_deleted_wal_generation() -- the pre-connect refusal
that stops a second opener from minting a replacement WAL under a live
writer -- was a permanent no-op on macOS. The reporter of #109641 hit
exactly that: after an update/restart took the sidecars away, a fresh
opener minted a new generation at the path, the still-live writer's next
write raised DeletedWalGenerationError, and each event copied the whole
database (16 halts / 13 minutes / 4.2 GB of captures).
macOS has no /proc and never reports a " (deleted)" suffix, which is why
the scan was restricted to Linux, but libproc does describe other
processes' descriptors: proc_pidinfo(PROC_PIDLISTFDS) lists a process's
fds and proc_pidfdinfo(PROC_PIDFDVNODEPATHINFO) returns each vnode fd's
(st_dev, st_ino) plus the vnode's last pathname -- for same-user
processes, without elevation. Both survive unlink, which is also why
psutil.Process.open_files() cannot stand in for it (it hides unlinked
descriptors, so the retired generation is structurally invisible).
The judgement itself is unchanged and now shared: a descriptor counts
only when it names a watched sidecar path while its identity no longer
matches what that path holds, i.e. _fd_is_truly_unlinked()'s identity
test (#108082), never a path suffix or a link count. Only the source of
that identity differs per platform -- readlink on /proc for Linux,
libproc for macOS -- and the darwin side resolves symlinks before
comparing paths because libproc reports the kernel's path
(/private/var/... where the caller opened /var/...).
Scope is this one function: the enumeration legs, the gate (Windows
still returns [] -- it cannot unlink a held sidecar) and the stale
docstring reason. Enumeration failures keep the existing fail-open
behaviour (logged at debug, no holders), and the new tests are marked
macos_only so the existing Linux-only ones stay untouched.
Cost, measured on macOS 26.4 (darwin 25.4.0) with 721 processes /
4383 vnode descriptors: ~20 ms per full enumeration, versus the Linux
leg's ~11 ms / ~4.4k syscalls measured in #108910 -- the same order,
paid once per open, on the platform where the guard previously did
nothing at all.
(cherry picked from commit f1501dfe7141e3c2521c5192857c37d7b60a922b)
SQLite protects a WAL generation with per-PROCESS POSIX locks (SHARED on
state.db, DMS byte on -shm). Any in-process open()/close() of either file
cancels both (sqlite.org/howtocorrupt.html §2.2); the next last-connection
close in ANY process then checkpoints and unlinks -wal/-shm, and the holder
sticky-halts with DeletedWalGenerationError. #109841 removed one such
close (mode tightening) but the class is open-ended: raw header probes,
plugins, tool reads of ~/.hermes, any library that touches the files.
hermes_state_lockguard re-holds the same two ranges as OFD locks
(F_OFD_SETLK) on private descriptors for as long as a writer handle is
open. OFD locks belong to the open file description, so a stray close()
cannot cancel them, and they conflict with the EXCLUSIVE a sibling needs
for the close-time reset exactly like SQLite's own. Released before the
handle's own close so a true last close still ends the generation; the
descriptors are closed only once no connection to the path remains, so a
holder scan from another process never counts them. Works on Python 3.11
(where sqlite3 cannot arm SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE) and on macOS
(F_OFD_SETLK=90 per XNU bsd/sys/fcntl.h); no-op on Windows.
Live repro (Linux, Python 3.11.15, SQLite 3.53.1): holder = SessionDB
writer; in-process os.open/os.close of state.db and -shm; then a foreign
sqlite3.connect()+close(). Before: -wal unlinked, holder write raises
DeletedWalGenerationError. After: -wal keeps its inode, holder writes.
Applying a reasoning/speed default in Desktop Settings -> Model reset an
auxiliary slot a user had pinned via CLI back to provider "auto" / model ""
while leaving reasoning_effort intact (#95460). POST /api/model/set was
never the writer; writeAgentDefault was: it round-tripped the whole
default-expanded config record (loaded when Settings opened) through
PUT /api/config, so every key another surface changed since the snapshot was
echoed back with its stale, default-filled value. The pinned slot's
provider/model existed only as defaults in the snapshot; reasoning_effort was
already in it, hence the asymmetry the report observed.
PUT /api/config deep-merges onto disk, so a writer only needs to send the key
it changed. Every desktop single-key writer now does exactly that (the
config-settings page already diffed against a baseline): Model defaults
(agent.reasoning_effort / service_tier), Appearance resume_last_session,
terminal font, session auto-archive, the two browser.use_real_profile
toggles, and the Capabilities voice fields (diffConfig against a baseline).
The optimistic shared-cache write keeps the full merged record so sibling
surfaces repaint without a refetch. The dashboard's ReasoningPicker had the
same read-modify-write shape and now sends the sparse patch too.
Tests pin the wire contract: only the edited key is sent, a sibling pin that
is not in the snapshot cannot be echoed back.
The review fork is a full AIAgent, not an auxiliary_client call, and its
routed branch deliberately skips the parent's reasoning_config (the parent's
effort vocabulary may be invalid for the routed provider). It also never read
the per-task key, so an explicit `auxiliary.background_review.reasoning_effort`
was silently ignored and the routed fork ran at the provider default (#94825).
Routed forks now parse the task key through the shared parse_reasoning_effort
(same levels and `none` alias as every other aux task); unset keeps the
provider default, an unknown level warns and falls through. The same-model
path is untouched: it still inherits the parent's reasoning_config verbatim
for prompt-cache parity.
Salvaged from #94832 (liuhao1024), re-applied on the decomposed
_fork_init_kwargs seam.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
auxiliary.background_review.reasoning_effort was a silent no-op on the
same-model path: the fork inherits the parent's reasoning_config verbatim to
keep prompt-cache parity (#30532), and nothing told the user. Emit a one-time
user-visible warning (parent-scoped, so a nudge-per-turn session warns once,
not per fork) when the key is actually set, document the no-op in the config
reference, and leave the fork-birth request bytes unchanged.
The Desktop AUX_TASKS registry drifted from web_server._AUX_TASK_SLOTS:
triage_specifier, kanban_decomposer, and profile_describer were served by
the backend (stale-aux warnings referenced them) but had no row in
Settings > Models > Auxiliary models, leaving "Reset all to main" as the
only way to change them. Add the three rows with localized labels/hints
(en/ar/ja/zh/zh-hant) mirroring the CLI _AUX_TASKS descriptions, and pin
the rendered set with test assertions.
Fixes#97297
The rebased synthetic guard built a bare dict that main's _build_payload
no longer accepts (it indexes aspect_ratios/resolutions directly) and never
exercised generate(); it now builds the family via _family() and asserts
generate() returns modality_unsupported without submitting. The surface
matrix's i2v-only parametrization collected zero cases after Gemini Omni
Flash 1.1 gained t2v, so it is removed along with the dead branch in the
t2v matrix.
FAL shipped google/gemini-omni-flash/v1.1/* (Aug 2026): the family gains a
text-to-video endpoint, 360p/720p/1080p/4k resolution enum, and keeps
3-10s integer durations with always-on native audio.
- plugins/video_gen/fal: bump gemini-omni-flash to the v1.1 endpoints,
declare the resolution enum, refresh display/strengths copy
- tests: family test asserts the versioned dual-modality endpoints; the
i2v-only clean-error guard survives via a synthetic family; catalog
invariant now requires both endpoints on every family
Schema verified against FAL OpenAPI (t2v: prompt required, 16:9/9:16,
360p-4k, duration 3-10 int; i2v adds image_url + optional end_image_url).
Pricing: $0.03/s 360p, $0.10/s 720p, $0.15/s 1080p, $0.30/s 4K.
_attach_child now mirrors a pending parent stop with the same split
interrupt() uses for its own fan-out (hard -> hard_interrupt, soft ->
interrupt), so a redirect is not turned into a cancel on a child that was
attached late. _restore_parent_cancellation collapses to re-attaching the
rejected unit's children: the replay is the attach step's job now.
Test fixture: _Batch gained origin_session_history_delivery on main after
the salvaged PR was written.
Co-authored-by: illidan <noequal666@gmail.com>
Fold the whitespace-only case into the empty-file miss test and drop
test_overwrite_is_all_or_nothing: it wrote sequentially, so it exercised
nothing about os.replace atomicity that the publish test does not.
CI runs the suite as ~96 concurrent per-file pytest subprocesses and does
not install filelock, so the adapter-guard cache in
tests/gateway/conftest.py runs lock-free there. The verdict was published
with Path.write_text (open-truncate-write): a sibling reading between the
truncate and the write saw an empty file and raised pytest.UsageError(""),
failing a healthy tests/gateway/relay/* file with a bare 'ERROR:' line
(flaky-retry frames in CI runs 32930243538, 32932008269, 32933117564).
Fix the class, not the window:
- _write_guard_cache_atomic: publish via staging file + os.replace so
readers see the old state or the complete verdict, never a torn write.
Staging name (.tmp-*) is invisible to the stale-fingerprint eviction
globs so a sibling's eviction can't unlink it mid-publish.
- _read_guard_cache: empty/vanished cache is a miss (rescan), never a
violation verdict.
Repro: 24 concurrent lock-free subprocesses over 3 relay files fired the
empty-ERROR failure 8x in 8 rounds on origin/main; 0x in 10 rounds after
the fix. Regression tests sabotage-verified.
The Python-repr pass ran after the MCP probe header scrub and collapsed
'Authorization': 'Digest ***' to '***', erasing the scheme word that scrub
deliberately keeps (tests/hermes_cli/test_mcp_probe_redaction.py went red
on rebase). Skip values that already carry a mask marker.
Port from OpenHands/software-agent-sdk#4508: their dict-entry secret
redaction was uppercase-only and leaked mixed-case keys (UserPassword,
sessionToken). Apply the same case-insensitive treatment to the Python
mapping-repr pass: a casefolded credential suffix (apikey/token/secret/
password/passwd/credential) now qualifies a key, while metadata names
(TOKEN_COUNT, password_policy, tokenizer) stay untouched.
A /p/<profile>/ cron_job route resolved and fired the job from the gateway's
default home: `_fire_cron_job` ran `execute_job_for_event` outside
`_profile_scope`, so `cron/jobs.json` lookups (`resolve_job_ref`,
`claim_job_for_fire`) and the run's config/secrets came from the wrong
profile — the agent-mode path already scopes its run, this path did not.
`asyncio.to_thread` copies contextvars, so wrapping the call is enough.
Route-level `skills` were also being injected into the rendered prompt on
cron_job routes even though the docs say they are ignored (the job's own
skills apply); skip `_apply_skills` for cron_job routes so the per-run
context stays plain event text.
Test proven red on the pre-fix tree (home resolved to the default profile),
green after.
ChatGPT Work's Aug 25 2026 release lets scheduled tasks fire from app
events (new Gmail message, Slack activity, GitHub PR feedback) instead
of polling on a cadence. This ports the pattern by composing two
existing Hermes subsystems: a webhook route can now set cron_job to
fire an existing cron job on each inbound event.
- gateway/platforms/webhook.py: cron_job route mode — after the same
HMAC auth / rate limit / filters / script / idempotency as agent
routes, the rendered prompt becomes transient per-run context and the
job fires through execute_job_for_event on a worker thread (202
Accepted immediately). Startup validation rejects cron_job +
deliver_only.
- tools/cronjob_tools.py: execute_job_for_event() — public wrapper over
the shared claimed-run body (_execute_job_now), so event fires share
at-most-once claiming, in-flight dedupe, delivery, and [SILENT]
handling with scheduler and manual runs.
- hermes webhook subscribe --cron-job: creates event-trigger
subscriptions; job ref validated (and canonicalized to the job ID) at
create time.
- Docs: webhooks.md route table + Event-Triggered Cron Jobs section,
cron.md capability list, zh-Hans mirrors.
- Tests: tests/gateway/test_webhook_cron_trigger.py (adapter + unit),
CLI tests in test_webhook_cli.py.
Port from openclaw/openclaw#120597: Docker Desktop, OrbStack, and Podman
expose host bind mounts as fuse.virtiofs or 9p. SQLite WAL shared-memory
is not coherent across the VM boundary and fails SILENTLY under write
pressure (zero-filled pages), so the existing reactive marker fallback
(_WAL_INCOMPAT_MARKERS) never fires — the mode must be refused before
the pragma.
apply_wal_with_fallback now checks the DB path against
/proc/self/mountinfo (longest mount-point prefix wins; per-directory
cached) and applies journal_mode=DELETE with a one-shot WARNING naming
the fix (native volume instead of a bind mount). On-disk WAL databases
are still never live-downgraded; require_wal callers get
WalUnsupportedError. Non-Linux hosts and unreadable mount tables
conservatively skip detection (behavior unchanged).
Sabotage-verified: disabling only the wiring makes
test_fresh_db_on_cross_vm_fs_gets_delete fail.
audio_param_key is a new _build_payload seam; one invariant covers it (Wan
emits `audio`, veo3.1 still emits `generate_audio`) plus start_image_url
and the integer duration.
Alibaba's Wan 3.0 generation went live on FAL (was "Coming Soon" since
the Wan 2.7 rollout). Adds two families to FAL_FAMILIES:
- wan-3.0: alibaba/wan-3.0/{text,image}-to-video — 2-30s integer
durations, 480p/720p/1080p, native audio, adaptive aspect handling
($0.05-$0.20/s by resolution)
- wan-3.0-prime: alibaba/wan-3.0-prime/{text,image}-to-video — premium
tier, same surface ($0.068-$0.28/s)
Schema quirks mapped from the FAL OpenAPI/llms.txt:
- i2v takes `start_image_url` (image_param_key, Kling-4K pattern)
- duration is a JSON integer (duration_int)
- audio toggle key is `audio`, not `generate_audio` — new generic
`audio_param_key` family flag in _build_payload (veo3.1 regression
asserted unchanged)
The streaming accumulator dropped delta.reasoning_details entirely — only
non-streaming responses preserved the OpenRouter unified reasoning replay
data (signatures, encrypted reasoning blocks). Providers that require the
reasoning_details sequence passed back on the next turn (OpenRouter
reasoning models, Anthropic signed thinking via shims) lost continuity on
every streamed turn, which is nearly all turns.
- Accumulate delta.reasoning_details (attr or model_extra) during
streaming and attach the merged list to the final mock message, where
_build_assistant_message's existing passthrough persists it.
- _append_streamed_reasoning_detail merges consecutive reasoning.text /
reasoning.summary fragments into one logical entry (OpenRouter streams
them as word-level deltas; unmerged they bloat the replayed signature
payload) while encrypted/opaque entries stay discrete. Later fragments
backfill signature/id/format/index fields the first fragment omitted.
Ported from earendil-works/pi#8605 (commit c5ad7c1b0), credit
@cristinaponcela for the merge-fragments pattern.
Tests: tests/run_agent/test_streamed_reasoning_details.py (8 tests;
sabotage-verified: disabling accumulation fails the E2E test, disabling
merging fails 4 unit tests). Neighboring suites green (test_streaming.py
40 passed).
Rebased onto main (~9700 commits): the branch predates the god-file
decomposition and the tests/state -> tests/hermes_state move.
Fixes for seams that moved:
- cron memory contract patches cron.scheduler_delivery._resolve_origin and
hermes_state_registry.acquire (where run_job now reads them).
- state.db conformance imports _live_writer_holds_db from
hermes_state_repair and drives the public create_quick_snapshot.
- update receipt: serve runtimes are reconciled in their own unit
vocabulary since #100479, so the "full accounting" row names the serve
unit instead of borrowing a gateway relaunch.
Deleted (change-detectors / source-greps / duplicates / dead code):
- source-text scan of hermes_state*.py for "maintenance-shaped" defs and
the symbol REGISTRY it fed (renames are not regressions).
- per-op exact-outcome table under a live writer -> one invariant: refuse
with a lock error or report zero work, DB stays intact.
- copy_db_and_verify pins: the symbol is a revert-scheduled plugin-compat
pointer, not production code (check_compat_pointers.py).
- cron ON-direction tests already pinned by tests/cron/test_scheduler.py,
plus a tautology that never called production code.
- exact warning-wording asserts in the env deprecation truth table.
- same-file duplicates: fixed-point (implied by idempotence), tripwire
round-trips, absent-registry, sequential "race" re-enactment.
Two weeks of closed issues/merged PRs show the same areas regenerating:
each salvage pinned its instance while the class invariant had no test.
These suites pin the invariants themselves:
- tests/conformance/test_profile_write_tripwire.py — no writes to the
default profile tree while a profile is active (#88532#92662#89190#89625#92156); reusable tripwire fixture, 4 surfaces
- tests/hermes_cli/test_env_deprecation_truthtable.py — 18-row truth
table for the Deprecated-.env warning (#88829#89016#89389#90299)
- tests/cron/test_cron_memory_contract.py — cron<->memory contract that
flipped twice in Aug (#91269 -> #91384 -> #91447)
- tests/agent/test_injected_param_strip_retry_registry.py — every
strippable injected param x real 400 shapes must strip-and-retry;
unknown params must still fail (#90257#89897#91164#89503)
- tests/agent/test_transcript_decoration_idempotence.py — f(f(x))==f(x)
law + 4-breakpoint budget for apply_anthropic_cache_control (#90971)
- tests/state/test_state_db_maintenance_conformance.py — registry-
enumerated maintenance ops refuse/degrade under a live writer; copies
of corrupt DBs are refused or flagged (#91839#90806#90613#88235)
- tests/tools/test_bot_mode_canonical_chat_resolution.py — canonical
Bot Chat resolution is idempotent, never mints, unique per profile,
race-safe (#92040#90705#92692#90005#90732, PR #92129)
- tests/hermes_cli/test_update_receipt_truthfulness.py — receipts:
crash never claims success; success requires full fleet accounting;
refusal != failure (#91283#91439#92902#92780)
117 tests, all sabotage-verified (each suite proven to FAIL when its
bug class is reintroduced).
The shell `find ~/.hermes/skills ~/.hermes/hermes-agent/optional-skills ...`
snippet hardcoded a dev-clone path that does not exist on user installs; the
loader already substitutes ${HERMES_SKILL_DIR} (skills.template_vars), which
is what every other bundled skill uses. Drop the upstream-agent path mention.
Regenerated the docs page so it matches SKILL.md (also fixes the comfyui
related-skill link, which now lives under optional).
Optional skill: scroll-as-timeline landing pages on a deterministic
CSS/JS engine, with interview → page grammar → signature move workflow
and screenshot-based scroll verification. Engine and scripts vendored
verbatim; asset generation re-anchored on image_generate with the
upstream kie.ai flow kept as an optional path.
- The Kanban plugin cannot import `@/lib/ime` (plugin fence), so its
`./ime-enter` twin duplicated the predicate. Export `isSubmitEnter` from
`@hermes/plugin-sdk` and drop the twin so one helper owns the policy.
- `BoardNameField` in kanban/board-switcher.tsx (main's refactor of the
create/rename board dialogs) submitted on composition Enter; guarded.
- Telegram allowed-ID input and the clarify-card textarea submitted on the
post-compositionend keyCode-229 Enter; both now use `isSubmitEnter`.
Widens the salvaged Kanban fix (PR #94611 by @huklaa) to the whole bug
class, following cloudflare-os PR #291 which centralized one
isImeComposing predicate and swept every Enter-submit site.
- New shared helper src/lib/ime.ts (isImeComposing / isSubmitEnter):
handles nativeEvent.isComposing (React), event.isComposing (DOM), and
the legacy Chromium/Safari keyCode 229 commit-Enter that arrives after
compositionend.
- Sweeps 21 previously unguarded Enter-submit sites: dialogs (project,
worktree, profile-remote-override, pet rename), settings fields
(credential keys, model API key, quick-entry shortcut, attachment
size, combobox), quick entry, pet overlay composer, pet generate +
hatch, review ship bar, file rename, preview browser address bar,
model catalog menu, MCP setup + approval strips, Kanban drawer.
- Upgrades two partial guards (onboarding, session-actions-menu) that
checked isComposing but missed keyCode 229.
- find-in-page: Enter step no longer fires mid-composition (typed CJK
search queries jumped the viewport on every conversion commit).
Validation: tsc clean, eslint clean, 135 tests green across ime/find-bar/
composer/kanban suites; sabotage run (guard removed) fails the new test.
Pasting exactly one http(s) link while composer text is selected now
turns the selection into a markdown link ([selected text](url)) instead
of replacing it — the behavior every rich text editor ships.
- resolveExactLinkPaste(): recognizes a clipboard payload that is exactly
one supported link (bare or <...>-wrapped, host required, no prose or
trailing punctuation) and returns the href.
- selectionLinkLabel(): the selected composer text eligible for linking;
rejects collapsed selections, selections spanning ref chips or line
breaks, and whitespace-only selections.
- markdownLinkFor(): builds the markdown link, escaping square brackets.
- handlePaste wires the three together ahead of the @url: chip path;
any non-qualifying paste falls through to existing behavior.
Adapted from Buzz's TipTap link-mark approach to Hermes' contenteditable
composer: we emit a markdown link (the composer's native rich construct)
rather than a ProseMirror mark.
The prompt template is built from _PROMPT_GOOD_EXAMPLES, so asserting the
template contains them can never fail independently of the code it checks.
The behavioural invariants (echo rejected, greeting allowed, near-miss
passes) stay.
Xipong