Commit Graph

15357 Commits

Author SHA1 Message Date
teknium1 02005cfe20 fix(kanban): promote refuses undone parents instead of a false --force success
`hermes kanban promote --force <id>` printed `Promoted <id> -> ready` and
then the very next claim (a human `claim`, or the dispatcher tick seconds
later) demoted the task back to `todo` with `claim_rejected
{parents_not_done}` and returned None (#106195). The non-force refusal
even pointed operators at `--force` as the escape hatch.

The claim gate is deliberate: `claim_task` is the single enforcement point
("never ready -> running with an undone parent, whichever writer set
'ready'", cda20eec0c), and `complete_task`/`request_review` re-check the
same predicate, so a child let through by a forced claim could still never
finish. A promotion override therefore has no honest outcome; the
dependency edge is the real knob.

- drop `--force` from `promote` (parser, CLI handler, `promote_task`
  kwarg, the `forced` event field nothing read)
- the refusal message now states why the gate cannot be bypassed and names
  the working remedies: complete the parents or `hermes kanban unlink`
- two invariant tests: refusal on an undone parent leaves `todo` with no
  fake `ready`; the flag no longer parses

Salvage direction from #75354 by @vyacheslavk (diagnosis of the promote ->
claim gap); the consume-at-claim authorization there is not taken because
the same parent gate also blocks completion of the forced child.
2026-09-09 09:21:29 -07:00
Teknium ae65399d8f test(cron): trim kind-flip repeat tests to the two invariants
Keep the two tests that fail without the fix (one-shot -> recurring drops the
implicit times=1, recurring -> one-shot gains it) and fold the explicit-repeat
case into the first; drop the same-kind and explicit-finite cases, which only
pin behaviour the fix never touches.
2026-09-09 09:20:20 -07:00
Konstantin Khlopkov 111aea1a01 fix(cron): re-derive repeat defaults when a schedule update flips the kind 2026-09-09 09:20:20 -07:00
Totoro-qaq 5d6d5fb223 fix(cli): refresh TERMINAL_CWD when --in re-homes the session
`--in DIR` only chdir'd. Every cwd consumer (resolve_agent_cwd -> Codex
app-server thread cwd, the terminal tool, context-file discovery) prefers
TERMINAL_CWD over the process cwd, so a value inherited from a parent
Hermes surface, the shell or .env survived the chdir and the session kept
running in the old directory. The local backend was rescued by cli.py's
force-export at import time; docker/ssh backends and the TUI launch path,
which never imports cli.py, were not.

Refresh TERMINAL_CWD to the --in target when it is already set. An unset
variable stays unset so the backends keep deriving from the new process
cwd and no host path is pre-seeded into ssh/container backends.

Fixes #106220
2026-09-09 09:20:12 -07:00
teknium1 060cd7f9bb fix(state): trim the futile-holder FTS diagnostic to shape and fix the remedy text
Slim redo of the mechanism from #106410 on top of its pick (no wrappers, no
persisted "kind" enum, no process-local flag that dies with the process):

- Futility = the SAME holder PID set has blocked >= _FTS_HOLDER_FUTILE_ATTEMPTS
  (10) deferrals over >= _FTS_HOLDER_FUTILE_SECONDS (30 min); tracked as
  holders_since/holders_attempts in the persisted fts_rebuild_deferral record
  and reset whenever the holder set changes. The 3-deferral/60 s escalate
  window is the orphan-reap gate and stays as is.
- ONE escalated ERROR line names each holder pid + cmdline and the remedy that
  can actually be followed from inside a gateway session: stop ONLY the other
  holder; this process's own retry admits the rebuild within 60 s. The old
  "with the gateway stopped" advice was unrunnable from a gateway-hosted
  session (the gateway is the session) and is gone from both log and doctor.
- hermes doctor renders the futile record distinctly.
- retry_deferred_fts_recovery: a capped backoff earned by holder set X no
  longer applies once the live holder set differs from X, so stopping the
  other service is followed by a retry on the next tick, not up to an hour
  later (the issue's 16-min wait).
- Tests trimmed from 5 to 2 invariants (futile line + doctor entry after N
  same-holder deferrals; backoff reset when the holder set changes); the
  contributor's control tests for changing PIDs / orphan reap are covered by
  the existing test_repeated_deferrals_reap_inactive_orphan_then_rebuild.

The "canonical writes and LIKE search remain available" WARNING is kept
because it is true on origin/main: a stale open drops every FTS trigger, so
the messages INSERT succeeds (probed live with a real state.db + a second
process holding it). Writes fail only when a peer re-publishes triggers over
the corrupt index — a separate class, not this diagnostic.

Refs #106393
2026-09-09 09:19:57 -07:00
KoNit-K e7ca9b47ac fix(state): diagnose futile FTS deferral from a permanent holder
A supervised peer never satisfies the orphan reap, so stale-FTS repair retried forever with a misleading "canonical writes remain available" warning.

(cherry picked from commit e57f3a975d311aa44da1e92c5e727eba7c8cff70)
2026-09-09 09:19:57 -07:00
Teknium b8b2278440 test: import pytest in test_stderr_timestamp (marker needs it)
The spawns_gateway_lookalike marker was added to a module that never imported
pytest; collection failed with NameError in CI.
2026-09-09 09:19:54 -07:00
Teknium ca16cafee4 test(guard): block spawning a real gateway runtime from tests
Tests that exercise the dashboard's gateway-restart path can end up
spawning a REAL `python -m hermes_cli.main gateway restart` child when
the spawn seam is not intercepted. `_spawn_hermes_action` launches it
with start_new_session=True, so it outlives the pytest worker; the
child inherits the pytest-tmp HERMES_HOME, which is not a profile and
hashes to no service suffix, so `get_service_name()` resolves the
DEVELOPER's `hermes-gateway` unit, `systemd_restart` restarts the live
gateway, and without systemd the fallback runs `run_gateway()`
in-process forever and squats the webhook port.

Live repro on this machine (origin/main): an unintercepted spawn of
["gateway", "restart"] from a test restarted the production gateway
(MainPID 136820 -> 1689090, NRestarts=1). On 2026-09-03 a sibling
refactor moved `_spawn_hermes_action` from the `hermes_cli.web_server`
facade to `hermes_cli.web_server_gateway` ~10 minutes before the tests
were repointed; runs in that window patched a name production never
read and left 39 orphans alive for six days.

The live-system guard now rejects any subprocess primitive whose
command line the canonical matcher (`gateway.status.
_gateway_command_subcommand`) classifies as `gateway run|start|restart`.
Argv substrings are never consulted, so `gateway status`, `gateway
--help`, `hermes_cli.main serve`, etc. pass through. Three files that
deliberately spawn and reap a stub child with a gateway-shaped argv
(flock holders, sleep sleepers with an argv tail) opt out with the new
`spawns_gateway_lookalike` marker, which lifts only this check and keeps
os.kill guarded. Two canary tests pin the block and the pass-through.
2026-09-09 09:19:54 -07:00
Teknium 19f2f19987 fix(gateway): refuse to uninstall a systemd unit pinned to another HERMES_HOME
Defence at the exact boundary the incident crossed: systemd_uninstall() and
uninstall._remove_systemd_gateway() unlinked whatever get_systemd_unit_path()
returned. Before stop/disable/unlink, read the unit's own
Environment="HERMES_HOME=..." line (the parser status/refresh already use)
and, when it names a different home than this process, warn with both paths
and leave the unit alone. A unit without the line (hand-written) is still
removed as before.
2026-09-09 09:19:36 -07:00
Teknium 4746e34448 fix(gateway): foreign HERMES_HOME no longer resolves to the default hermes-gateway unit
_profile_suffix() compared HERMES_HOME against get_default_hermes_root(),
which treats ANY home outside ~/.hermes (Docker /opt/data, a mktemp dir) as
"the root itself". Every such home therefore collapsed to the bare
`hermes-gateway` service name and the default profile's unit path
(~/.config/systemd/user/hermes-gateway.service); the documented
"else a short hash of the path" branch was unreachable.

A parity harness run with HERMES_HOME=$(mktemp -d) called
uninstall_gateway_service(), resolved to the production unit, ran
`systemctl --user stop/disable`, unlinked it and daemon-reloaded. With the
unit gone Restart= could not revive it: all cron jobs and every messaging
platform were down for 6.5 days.

Compare against the platform-native default home (~/.hermes) for the bare
name; keep the profile name for <root>/profiles/<name>; everything else
(temp dirs, Docker /opt/data) gets its sha256[:8] suffix as the docstring
always promised. The Docker image supervises with s6 (`gateway-<profile>`
slots), not systemd/launchd, so the bare host-service name was never load-
bearing there.
2026-09-09 09:19:36 -07:00
kshitijk4poor f3ae63162e test(auth): keep the persisted clean-mark tests to the two invariants
A fresh process with an unchanged store takes zero auth-store locks, and a
root store that gains a forked grant invalidates the persisted mark so the
heal re-runs. The corrupt-mark fallback, same-mtime size change and
no-secrets checks were pinning implementation details of the same cache.
2026-09-09 21:17:35 +05:30
John Paul Soliva 57e03e2be6 perf(auth): persist the forked-OAuth clean mark so a fresh process skips the heal's locks
`_heal_forked_single_use_oauth_grants()` runs on every `load_pool()`, and its
clean mark lived only in memory. Every fresh `hermes` invocation and every new
worker therefore re-took the heal's two nested EXCLUSIVE auth-store locks just
to rediscover a store it had already cleared — 4 acquisitions per process on a
two-provider profile, every one of them finding nothing to consolidate. Behind
a sibling process holding those locks that costs a full
`AUTH_LOCK_TIMEOUT_SECONDS` per provider before the process can do anything at
all: measured 30.1s for two providers.

Persist the mark next to the store it describes (`<profile>/cache/
oauth_heal_clean.json`, 0600, paths and stat data only — no credential
material) and consult it BEFORE taking any lock.

Outliving the process means the mark needs a stronger key than the in-memory
one did:

- The ROOT store joins the fingerprint. This heal consolidates root → profile,
  so root acquiring a counterpart turns a row the heal deliberately KEPT into a
  fork it must strip. The in-memory mark could ignore root because it died with
  the process; a persisted mark would keep skipping a heal that has become
  necessary.
- File sizes join it too, so a metadata-preserving rewrite (`rsync -t`,
  `tar -p`, a restore) cannot leave a stale mark looking current indefinitely
  rather than for one process.

Measured on an isolated HERMES_HOME with two OAuth providers:

    lock acquisitions per fresh process    4     -> 0
    contended load_pool() x2               30.1s -> 0.00s
    mark-file reads per 100 load_pool()    -     -> 2 (one per provider)

The mark stays a cache: absent, unreadable, corrupt or wrong-shaped content all
mean "unknown" and fall through to the locked heal, and a failed write only
means the next process re-runs it — the behaviour before this cache existed.
2026-09-09 21:17:35 +05:30
kshitijk4poor e2bd400233 fix(models): only cache unreachability, not HTTP errors; key the entry via base_url_origin
An HTTPError means the host answered — a 401 from a wrong API key must not
be remembered as "unreachable" for the next 60s, or a user who fixes the key
gets a cached empty catalog on the immediate re-probe. Connection-level
failures (timeouts, refused, DNS) are the only thing the cache records.

_probe_neg_key hand-rolled scheme/port defaulting that utils.base_url_origin
already provides; use it.
2026-09-09 21:16:48 +05:30
kshitijk4poor 20762c67eb test(models): trim probe negative-cache tests to two invariants 2026-09-09 21:16:48 +05:30
finn763 48b8528e7c fix(desktop): stop UI freeze on unreachable provider Closes #81123 2026-09-09 21:16:48 +05:30
kshitijk4poor a4113eb994 test(dashboard): keep the read-coalescing tests to the two admission invariants
The contributor's 15-test module pinned encoding details (frozen-arg shapes,
postponed-annotation resolution, monkeypatch seams). The two behaviour
contracts that matter survive: a 12-request /api/profiles burst admits ONE
worker and leaves /api/status responsive on a 2-token pool (this file), and
the kanban board read shares one worker per key (test_kanban_read_admission).
Both go red when the coalescing wrapper is removed from the route.
2026-09-09 21:16:40 +05:30
Gianpietro Dal Zio 653418d842 fix(dashboard): coalesce expensive reads before worker admission 2026-09-09 21:16:40 +05:30
kshitijk4poor 441157f184 test(models): two invariants for the OpenRouter curated-catalog disk cache 2026-09-09 21:16:28 +05:30
finn763 a74e76632c perf(picker): read Nous pricing cache-only when building the picker row
_nous_picker_model_ids only uses the ids the Portal unions append — both
unions discard the pricing map (`model_ids, _ = union_with_portal_*`) — yet
it called get_pricing_for_provider("nous") without cached_only, so a cold
pricing cache paid a full /v1/models round-trip (network timeout on a slow
Portal) on the picker-open path for nothing. Pass cached_only=True; the
background pricing prewarm (#101685) fills the same cache for later opens.

Re-derived from #102099 by @finn763: the original patched
hermes_cli/model_switch.py, which 3b1ecfc0a1 decomposed; the live call site is
hermes_cli/model_switch_providers.py.

Based on #102099 by @finn763.
2026-09-09 21:12:22 +05:30
kshitijk4poor b36f654271 test(profiles): pin projects/tree single-flight as a behaviour contract
Replace the attribute-presence check (asserts a cache_clear attribute the decorator does not
expose) with one invariant: eight concurrent projects/tree requests run the profile scan once
and each caller gets its own copy of the payload. The scope-suite fixture now disables the TTL
instead of calling a non-existent clear hook, which silently no-op'd.
2026-09-09 21:11:56 +05:30
moken627-hub 35af06c009 fix(desktop): coalesce projects/tree sidebar scans and memoize raw config parses
The all-profiles sidebar polls GET /api/profiles/projects/tree, the one
heavy sidebar endpoint that was not wrapped in @_sidebar_singleflight_cache.
Every poll fanned out list_profiles() + _build_project_tree() over every
profile (51 on this box, ~17k SKILL.md files walked), and each profile
resolution re-parsed its config.yaml because read_user_config_raw() ran
uncached. On a 2-vCore VPS running 'hermes serve' for the Desktop remote
backend this pinned both cores (py-spy: 110-116% sustained, HostHighCPU).

- Wrap get_profiles_projects_tree in the existing single-flight cache,
  matching get_profiles_sessions_sidebar (5s TTL, errors[] not cached).
- Memoize read_user_config_raw() on a (st_dev, st_ino, st_size,
  st_mtime_ns) fingerprint under _CONFIG_LOCK, same strategy as
  read_raw_config(). Hits return a deepcopy so write-back round-trips keep
  their fresh-dict semantics; parse errors are never cached; the docstring
  no longer claims 'no caching'.
- Tests: memo semantics (deepcopy isolation, inode-replace reparse, error
  non-caching, parse-count), a wrapper-presence pin for both heavy sidebar
  endpoints, and a cold-cache autouse fixture in the scope tests (the 5s
  TTL otherwise leaks one test's payload into the next).

Measured on the affected box: list_profiles 1.9s -> 0.16s warm, serve CPU
112% -> 22-34%, host CPU 50-70% -> ~26%.
2026-09-09 21:11:56 +05:30
kshitijk4poor ead7e91dab refactor(recovery): stream the salvaged population; table the shape rules
- Pass 1 no longer materialises every classified record (full
  `messages.content` included) until pass 2; `LayoutEvidence` keeps only the
  capped per-position value sets (+ sessions rows for the one cross-column
  invariant) and pass 2 re-streams the lost_and_found tables. A 276 MB
  corrupted store no longer has to fit in memory.
- `_sentinel_holds` / `_text_shape_holds` if-ladders become rule tables.
- Tests trimmed to the three that bind behaviour (upgraded store maps by
  name; verifier refuses when rows matched no layout; replayed history ends
  at the current schema — the drift guard). No behaviour change; reverting
  inference to "no layout" still fails the name-mapping test.
2026-09-09 18:28:57 +05:30
kshitijk4poor 17c43aba06 fix(recovery): infer the salvaged store's physical layout from its schema history
A store's physical column order depends on which schema it was created at and
which ALTER TABLE ADD COLUMNs it lived through; one hardcoded "upgraded"
layout cannot cover them. `session_schema_history` records the declared
schema of the salvaged tables over time; `reachable_physical_layouts`
replays it to enumerate every physical order a store can have. The mapper
infers the layout once per kind from the whole recovered population (one
store wrote all of them), maps cells by name, and counts records whose width
matched no layout; the recovery verifier refuses to report such a salvage as
healthy.

Rebased onto the simplified session modules (c88d60551e, b9b4600cb2,
7e5a1a11d9, 1915a0d27e); no behaviour change from the pre-rebase branch.
2026-09-09 18:28:57 +05:30
Justin Wilson 7704712168 fix(sessions): map lost_and_found cells by physical column names
Upgraded state.db files gain columns via ALTER TABLE ADD COLUMN, so
physical order diverges from SCHEMA_SQL. Prefix-mapping onto the fresh
template put started_at at 0 and shifted titles/models. Insert by name
using known physical layouts; keep the plausibility gate.

Fixes #101409
2026-09-09 18:28:57 +05:30
kshitijk4poor a7f2a593d1 fix(db): honour require_wal on the probe-unknown path; touch nothing at all
Follow-up to the probe-unknown guard:

- `require_wal=True` now raises WalUnsupportedError when the on-disk mode
  cannot be read instead of reporting an unverified "wal" (the function's
  contract is "the mode actually set").
- Drop `_apply_wal_companions` from the branch: journal_size_limit /
  synchronous pragmas were being applied to a file whose mode is unknown,
  contradicting the "touch nothing" rule the branch exists for.
- Cut the retracted #104596 mechanism narrative from comments, log text and
  docstrings; the guard is hardening (same rule the DELETE branch already
  applied), not a root-cause fix.
- Tests: one binding test per file (no pragma of any kind reaches the
  connection; require_wal raises); drop the log-dedupe test (`_log_once`
  is already covered). test_captures_cause_on_failed_init's double now
  fails only set-pragmas, matching the read-only-mount it simulates.
2026-09-09 18:26:15 +05:30
webtecnica d0653ab090 fix(db): never emit WAL set-pragma when the on-disk probe fails (#104596) 2026-09-09 18:26:15 +05:30
Ahmett101 dc8044c086 fix(state): quarantine a state.db whose page 0 is not SQLite, not just a zeroed one
Startup only quarantined a 0-byte / all-NUL state.db. A file whose first page
was clobbered with record bytes (#102198) went straight to sqlite3.connect,
which raised "file is not a database" and deleted the -wal sidecar — the one
piece of evidence that could have been recovered.

`has_invalid_sqlite_header_preopen` generalises the zeroed probe (zeroed is a
subset of "no SQLite header"; same live-connection contract, never raises).
`quarantine_invalid_state_db` moves the file AND its -wal/-shm aside as
`state.db.<zeroed|notadb>-<ts>-<pid>.bak` before anything opens it; a fresh
DB is created as before.

Re-authored on current main (the quarantine helpers moved to
hermes_state_dbfile.py in d15c61b5dc); one regression test proves the
notadb case is quarantined with its sidecars and the new DB passes
integrity_check.

Refs #102198 (the write-after-SIGTERM that clobbers page 0 is not addressed
here; this preserves the evidence instead of destroying it).
2026-09-09 18:25:11 +05:30
kshitijk4poor 9e6c4100cb fix(agent): close the interrupted tool tail on the overflow terminal
The overflow-terminal path ends the turn without reaching finalize_turn, so
a transcript that overflowed right after a tool batch ended on a raw tool
result; strict providers reject the next user turn (tool -> user). Close it
with the same final text, mirroring the truncated-tool-call terminal above.

Also: classify once before either log so an overflow no longer emits a
"so the loop can continue" WARNING followed by the contradicting "NOT
seeding" one; reset the stale-streak breaker once for both branches; drop
the "compression could not recover it" wording (this path never reached
compression); trim the test file to the three tests that bind behaviour
(stream -> terminal stub; 413 stays non-terminal; the terminal ends the
turn, closes the tool tail, carries compression_exhausted).
2026-09-09 17:45:12 +05:30
ca-shrimp 3b0e81459b fix(agent): carry compression_exhausted bit; scope overflow terminal to context_overflow
Address review P1s (andrexibiza) on #106266:

1. The overflow-terminal exit in recover_from_truncation now forwards the
   #98722 typed compression_exhausted bit (partial_result/end_turn gained the
   flag) so the gateway resets/moves future input to a clean session instead
   of leaving the bloated durable session authoritative for the next turn.

2. _overflow_terminal is scoped to FailoverReason.context_overflow ONLY.
   payload_too_large (413) has its own byte-scored recovery owner
   (turn_overflow._recover_payload_too_large, #88960/#47339) that must not be
   bypassed; a post-delta 413 keeps its normal continuation stub. Regression
   covers both lanes.

Tests: unit asserts result compression_exhausted=True on the marker; a
real streamed partial hitting a 413 payload-too-large error keeps content and
is not terminal. 50 streaming/continuation/gateway regressions pass.
2026-09-09 17:45:12 +05:30
ca-shrimp ece584b8f5 fix(agent): don't seed continuation stub after a context-overflow stream death
When a stream delivered text and then died on a context-overflow /
payload-too-large error, the partial content (often tens of KB) was seeded as
a length-continuation stub, growing the transcript monotonically. In a
session whose transcript cannot be compressed back under budget
(protect_last_n covers everything -> no_progress, or the summary would
itself be larger -> would_grow), every later request is larger than the one
that just failed — an unrecoverable loop where the user sees a 30+ minute
fake hang and the only remedy is killing the session (#106260).

classify_api_error already labels these errors context_overflow /
payload_too_large (should_compress=True). _partial_stream_stub now returns
an EMPTY stub marked _overflow_terminal for that class instead of seeding
the recovered text, and recover_from_truncation treats the marker as
terminal: the turn ends via the recovery contract with a clear message
(start /new) and the transcript is not polluted with the partial.

Normal partials (network stall, output-cap truncation, tool-call drops) are
unchanged — only the overflow error class changes behavior.

Tests: stub marker + empty content; a real streamed partial hitting a
'maximum context length' error returns the terminal stub; recover_from_
truncation ends the turn (no fragment/nudge appended) on the marker while a
normal stub still runs the continuation path. 67 streaming/continuation
regressions pass.
2026-09-09 17:45:12 +05:30
nftpoetrist 511633be90 fix(agent): stop the run-budget wrap-up notice from mutating a persisted tool row
_maybe_inject_run_budget_wrapup() appends its wrap-up notice to the newest
role:"tool" message in place, with no _DB_PERSISTED_MARKER check. Its sibling,
_maybe_inject_iteration_budget_warning(), got exactly this guard added in the
same recent saga (turn_iteration_prep.py), with the comment "an older turn may
already be cached."

The reachability is structural, not an edge case: _maybe_inject_run_budget_wrapup
is only ever called from prepare_iteration(), at the START of the next iteration
-- strictly after tool_executor.py's _flush_session_db_after_tool_progress has
already flushed and marked the previous iteration's tool row persisted. So every
successful injection was mutating an already-persisted row: the wire request for
that turn carried the notice, but the durable transcript never did, diverging
replay from the live bytes and invalidating the provider's prompt-cache prefix
from that row onward.

Fix:
- Add the same _DB_PERSISTED_MARKER guard to _maybe_inject_run_budget_wrapup,
  scoped to the specific tool row the reversed scan lands on (not just
  messages[-1], since this function -- unlike its sibling -- scans backward for
  the newest tool row rather than only checking the tail).
- Wire _maybe_inject_run_budget_wrapup into _flush_session_db_after_tool_progress
  (pre-flush), mirroring exactly how _maybe_inject_iteration_budget_warning is
  wired in both places. Without this, the guard alone would make the notice stop
  firing in the common case, since prepare_iteration's call site almost always
  hits an already-persisted row -- the pre-flush call site is what actually lets
  it land in durable bytes.

Verified empirically: read the real call graph (tool_executor.py's three
_flush_session_db_after_tool_progress call sites cover every tool-completion
path) to confirm the guard's premise, then added an end-to-end test using a real
AIAgent + SessionDB that flushes and checks the persisted row for the notice
text. Mutation-verified: reverting the two production files drops exactly the 2
new/updated assertions (28 pass, 2 fail); reapplying restores green (30 passed).
Also ran the sibling iteration-budget-warning and /steer suites (71 passed) to
check for interaction regressions -- none.
2026-09-09 17:04:30 +05:30
Teknium bcff0a920e test(cron): replace fail-closed drift tests with snapshot-as-pin invariants
The old tests encoded the skip (agent never constructed, [drift_skip] text,
alert-once bit). New invariants, proven red on origin/main: an unpinned job
with model_snapshot=A / provider_snapshot=P runs with AIAgent(model=A) and
resolve_runtime_provider(requested=P) after the global default moved to B/Q;
an explicit job pin still beats the snapshot; cron.model / cron.model_provider
still beat the snapshot; a legacy job without a snapshot follows the global
default. test_cron_drift_alert_once.py is deleted with the bit it tested; the
config-notice and impact-summary tests drop guard_enabled / model_drift_guard;
the desktop toast test asserts the informational copy.
2026-09-09 04:32:13 -07:00
Teknium 48465c3933 fix(profiles): --clone-all no longer copies cron jobs into the new profile
Cron jobs are scheduled work bound to the source profile and its origin
channel. A clone that inherited cron/jobs.json fired every job twice: two
gateways with identical job ids running the same weekly jobs in parallel
(double spend, duplicate deliveries) until one gateway died.

Root cause: `cron` was not in _CLONE_ALL_HISTORY_EXCLUDE_ROOT, so the
copytree in _clone_all_into carried jobs.json along. Add it to the
per-profile history exclude set (applies to any source, CLI, dashboard
and TUI/desktop RPC all funnel through create_profile), recreate the
_PROFILE_DIRS skeleton after the copy so the clone still has an empty
cron/ (and sessions/), and say so in the CLI summary line and docs.

--clone (config-only) never copied cron; export/backup keep cron as
before (an archive is a portable snapshot, not a second live profile).
2026-09-09 04:27:24 -07:00
Teknium e1838c5b5a fix: trim opencode-go 422 salvage to the invariant set
Drop the bare "tool.content" pattern (any 400 mentioning tool.content in a
non-list context would be sent through the image-strip path) and the profile
flag snapshot test; the behaviour tests (classifier verdict + proactive
downgrade) already pin the contract.
2026-09-09 03:52:47 -07:00
ericmaddox bee840bc8c fix(providers,agent): handle strict-string tool message validation and 422 on opencode-go (fixes #104731)
- Declare `supports_vision_tool_messages=False` and `supports_vision=True` on `opencode_go` provider profile in `plugins/model-providers/opencode-zen/__init__.py`
- Route HTTP 422 errors through `_IMAGE_TOOL_RULES` and add `tool.content.str`, `tool.content`, and `input should be a valid string` patterns to `_MULTIMODAL_TOOL_CONTENT_PATTERNS` in `agent/error_classifier.py`
- Add unit tests for OpenCode Go proactive tool result downgrade, HTTP 422 Console Go classification, and profile capability contract in `tests/run_agent/test_multimodal_tool_content_recovery.py` and `tests/plugins/model_providers/test_opencode_go_profile.py`
2026-09-09 03:52:47 -07:00
buihongduc132 c1ff9390f6 fix(backup): prune old hermes-backup-*.zip after each run, keep last 3
run_backup() previously wrote "hermes-backup-<timestamp>.zip" on every
invocation without deleting old ones. Hourly callers accumulated 157 zips
(14 GiB). Add _prune_run_backups() to keep the newest N (default 3,
configurable via backup.run_backup_keep or --keep CLI flag).
2026-09-09 03:33:14 -07:00
Teknium 734461d213 fix(models): same-URL custom endpoints stop evicting each other's cached catalog; no-probe picker opens revalidate
Two picker-freshness defects in cached_fetch_api_models():

1. The disk cache row was keyed on base_url only, with the credential
   fingerprint stored inside the row. N custom_providers entries sharing
   one proxy URL with different keys (#106184) took turns overwriting the
   single slot; every sibling then failed the fingerprint check, got an
   empty catalog, and disappeared from the Desktop pickers (which hide
   zero-model rows). Key on url#fingerprint so each credential owns a row.

2. cache_only opens (Desktop model.options without refresh) served a
   past-TTL row for up to 7 days without ever revalidating, so a model
   loaded on a non-current local endpoint stayed invisible until the user
   found "Refresh Models". Serve the stale row AND spawn the same
   off-thread SWR refresh the blocking path uses; the caller still never
   waits on the network.

Live repro (two rows, one URL, keys A/B; real loopback /v1/models):
  GUI no-probe open  before {'proxy-a': ['model-A1'], 'proxy-b': ['model-B1']}
                     after  {'proxy-a': ['model-A1','model-A2'], 'proxy-b': ['model-B1']}
2026-09-09 03:33:06 -07:00
briandevans f4c55323fa fix(cli): back up config.yaml before --reset overwrites it
`hermes setup --reset` calls `save_config(copy.deepcopy(DEFAULT_CONFIG))`,
which writes `get_hermes_home()/config.yaml` — the exact file the backup
block a few lines below copies to `config.yaml.bak.<timestamp>`. Because the
copy ran after the reset, the backup captured the defaults that had just been
written, not the user's config. The one invocation where a backup matters most
produced a worthless one, and the original was unrecoverable.

The block's own comment already claimed it runs "before setup modifies it";
on the --reset path that was false. Move it above the --reset branch so it
captures the true pre-setup state on every path.

Also report the backup location on the --reset path. --reset is destructive
and can leave the wizard early (the non-interactive return exits before the
end-of-setup notice), so a user who just lost their config was never told
where the copy is. The end-of-setup notice is unchanged for the normal path
and is suppressed only when it has already been shown, so no run prints it
twice; the shared wording now lives in one helper.

Behaviour otherwise preserved: `copy2` (config.yaml holds secrets, so mode is
preserved), the try/except fallback to `_backup_path = None`, and the existing
notice for the full-setup path.

Follow-ups deliberately out of scope: pruning accumulated `.bak.*` files, and
printing the notice on the other early-return paths (--portal, section runs).

Refs #3522
2026-09-09 03:32:58 -07:00
Teknium bf53ff00a7 fix(config): one bounded backups/config/ dir replaces four config.yaml.bak schemes
Four writers each dropped their own uniquely-named copy of config.yaml next to
the real file and none of them ever deleted anything: hermes setup
(config.yaml.bak.YYYYMMDD_HHMMSS, one per run even with no change), the
corrupt-YAML snapshot (config.yaml.corrupt.<ts>.bak), hermes migrate xai
(config.yaml.bak-pre-migrate-xai-<ts>) and the Docker boot migration
(config.yaml.bak-<ts>, .env.bak-<ts>). A home dir accumulated a dozen variants
with no way to tell which mattered.

hermes_cli/config_backups.py::backup_config is now the single writer:
backups/config/config.yaml.<reason>.<YYYYMMDD-HHMMSS>, skipped when the newest
copy for that reason is byte-identical, rotated to the newest five per reason.
backups/ is already excluded from full backups so nothing nests. Legacy
siblings written by the old schemes are moved into the dir on first use;
hand-named copies (config.yaml.bak-my-note) are left alone.

Live: three `hermes setup --non-interactive` runs against an unchanged config
went from three .bak files in HERMES_HOME to one pre-setup copy under
backups/config/; repeated loads of broken YAML produce one corrupt copy
instead of one per process (deduped by content).
2026-09-09 02:36:00 -07:00
Teknium 06dc51d62d fix: verification evidence ledger is inert while verify_on_stop is off
The ledger in verification_evidence.db exists only to feed the verify-on-stop
guard, but the recorder kept running on every foreground terminal command and
every file edit after #53552 turned the guard off by default. Users who never
opted in still accumulated a multi-MB database (7 MB / 4.6k rows on one install).

Every ledger entry point (record_terminal_result, record_verify_run,
mark_workspace_edited, verification_status) now checks verify_on_stop_enabled()
first and returns without opening or creating the database when the guard is
off. verification_status reports {"status": "disabled"} in that case; no client
consumes the verification.status RPC yet, so nothing downstream changes.

Existing ledger tests pin HERMES_VERIFY_ON_STOP=1 since they exercise the ledger
itself; the new test proves the off path never creates the file (red on base).
2026-09-09 02:35:41 -07:00
Teknium 677e8ed8a4 fix(desktop): SSH remote backend stops following the host's sticky active_profile
A Desktop-owned `hermes serve --isolated --ssh-session-token-file ...` child
is spawned with an explicit `--profile <name>` when the connection names a
remote profile, and with no flag for the remote root home. Without the flag,
`_apply_profile_override` read the remote host's sticky `active_profile`
file and re-homed the backend into whatever profile the user last selected
on that machine's CLI. Settings then read one config.yaml while the remote
gateway wrote another, so model picks and toggles "didn't stick".

Treat the SSH token flag as a fixed-identity marker, the same way
supervisor-launched gateway children are (#74872): a Desktop backend's
profile is chosen by the client, never by the host.

Live repro (before/after, temp HERMES_HOME with active_profile=foo):
  serve --isolated --ssh-session-token-file ...   hermes_home=<root>/profiles/foo -> <root>
  same + --profile foo                             hermes_home=<root>/profiles/foo (unchanged)
  serve (no token file, user CLI)                  hermes_home=<root>/profiles/foo (unchanged)
2026-09-09 02:34:59 -07:00
Teknium d0df324862 fix(tui-gateway): /review shows its reviewer in the Desktop subagent stack
`slash.exec` runs on the RPC pool, outside any turn, so `/review` dispatched the
reviewer with no HERMES_UI_SESSION_ID and no steer authority bound. delegate_task
registered the child with `owner_session_id=None`, `subagent.list` (owner-scoped)
returned nothing for the parent session, and the Desktop status stack's 5s
snapshot poll reconciled the live `subagent.start` row away — the user saw only
"Review started. Results will return here." with no subagent card.

Bind the same session identity a turn binds (`_set_session_context(...,
ui_session_id=sid)` + `_current_runtime_session_record`) around `start_review`,
and clear it after. The reviewer now registers under the parent sid with the
request transport as authority, so `subagent.list`, steer, stop and the Desktop
roster all see it.

Live repro (tui_gateway stdio, real OpenRouter reviewer):
  before: registry owner_session_id=None, owner_transport=NoneType;
          subagent.list -> {"subagents": []}
  after:  owner_session_id=<parent sid>, owner_transport=StdioTransport;
          subagent.list -> [{"goal": "Review recent work", "status": "running", ...}]
2026-09-09 02:00:21 -07:00
Teknium 13c580422c fix(auth): carry pool-row lineage into the provider-block heal
With account-identity matching gone, the providers.<id> block consolidation
only fired on shared token material. A historical fork (same copied pool-row
id, profile rotated, both pairs diverged) then healed the pool row into root
but left root's providers.openai-codex block on the spent pair; root's next
load_pool() re-seeds its device_code row FROM that block and undid the heal.

_HealPass now records that a profile pool row matched root by copied id or
shared tokens and passes that verdict to _heal_forked_provider_block, which
accepts it as lineage proof. No account-identity guessing is restored; an
independent same-account grant (no id/token match) is still left alone.

Follow-up to simpolism's #106177.
2026-09-09 01:46:03 -07:00
simpolism 73f9de0c3a fix(auth): preserve independent same-account OAuth grants 2026-09-09 01:46:03 -07:00
nftpoetrist 9e0dc4319a fix(state): guard vacuum() and optimize_fts() against quarantined SessionDB handles
A quarantined/replaced/split-generation handle must never run a full-file rewrite or an FTS5
'optimize': both read damaged or foreign pages and commit the result back, turning contained,
diagnosable corruption into an amplified one. Same guard _execute_write applies to every write.

Salvaged from #102092 onto current main: the _try_wal_checkpoint half landed via #106315's
_quarantine_reason(), so only the two rewrite sites remain.
2026-09-09 13:17:19 +05:30
kshitijk4poor 26f4a674e0 fix(agent): a /steer row is human input for every user-turn predicate
Follow-up to #106317. Typing the steer row (display_kind="steer") for the renderer and the
alternation-repair guard collided with the convention that any display_kind on a user row means
scaffolding: is_user_originated_turn / _is_actionable_user_turn / split_user_originated_turn
returned False for it (tail anchoring, auto-focus, dispatcher views, resume counts) while
_is_real_user_message returned True (anchor restoration) — the two predicate families disagreed
on the same row, and list_recent_user_messages (/undo, /rewind) skipped it in SQL. A steer
carries full user authority; the steer kind is now whitelisted in all four.

Also: the pre-API drain's requeue tail reuses _requeue_pending_steer instead of a copy; the TUI
history projection compares against STEER_DISPLAY_KIND; the steer() docstring describes the row.
2026-09-09 13:08:25 +05:30
kshitijk4poor bb2c961e9a fix(state): VACUUM is gated by the same quarantine rule as the checkpoints
Follow-up to #106315. vacuum() ran PRAGMA wal_checkpoint + VACUUM + wal_checkpoint(TRUNCATE) on
self._conn with no quarantine check; the only guard it inherited (optimize_fts raising
DeletedWalGenerationError) was swallowed by its own try/except and the rewrite proceeded on the
split-brain handle. Mutation on main: vacuum() returned 2 and rewrote pages after the write stop.
2026-09-09 13:06:34 +05:30
kshitijk4poor b530a482d8 fix(gateway): the ephemeral delete goes to the adapter that sent the final
Follow-up to #106316. send_final_ledgered resolved the live adapter internally and
_send_final_text resolved it a second time for _schedule_ephemeral_delete; a reconnect between
the two sent the delete to a transport that never owned result.message_id (the ownership rule
_final_delivery_adapter documents). The bracket now returns (result, adapter).

The queued lane carried the ledger identity through MessageEvent.message_id while the PR added
ledger_message_id for exactly that; it now uses the typed field, and the ledger read is
getattr-tolerant of duck-typed events (a missing attribute was swallowed as "ledger skipped").
2026-09-09 13:06:23 +05:30
kshitijk4poor 91433c8466 fix(loop): the turn-boundary export skips preflight-timeout envelopes and stops re-anchoring the persist index
Follow-up to #106312. _preflight_timeout_result carries the prior history without this turn's
user row (#7100); with a repeated prompt ("continue") the verbatim scan resolved to the
historical copy and exported it as this turn's proven boundary — the exact relabeling the export
exists to prevent. Nothing is exported for that envelope now.

The trailing `agent._persist_user_message_idx = idx` ran after finalize_turn had already flushed
the transcript, so it never influenced a persist and the next turn reset it: dead state, removed.
2026-09-09 12:55:43 +05:30
kshitijk4poor 7f3e0bb59a test(agent): the worker-start-failure test intercepts the callback worker again
`kwargs.get("target") is sched._run_callback` is always False (a bound method is a fresh object
per access), so the fake never returned Boom and the _dispatch failure branch went untested;
the test passed on the normal worker. Compare with == and assert the interception happened
(mutation: retiring the handle on start failure now fails the test).

The thread-count assertions sampled while per-fire workers were still live; quiesce every
handle with cancel(wait=) before sampling so the count is deterministic (AGENTS.md: timing tests
must not assume a quiet runner).

Follow-up to #106308.
2026-09-09 12:42:02 +05:30