`_sanitize_node` deleted the `required` key whenever the pruned list came
out empty. Four built-in tools (skills_list, todo, delegate_task,
session_search) declare `required: []`, so they left the sanitizer with no
key at all. Strict OpenAI-compatible proxies read the missing key as
`null` and 400 the whole request ("null is not of type array"), which is
non-retryable and kills the session on its first call.
An empty array is valid for every backend; the pruning was added (34c3e67)
to drop names that are not in `properties`, not to delete the key. Keep the
key with the filtered list, even when that list is empty.
Fixes#111684Fixes#59386
Co-authored-by: Cr4ckMe <jiqing.liu@whu.edu.cn>
scheduleReconnect parks the PTY as "closed" while the tab is hidden or the
chat route is inactive, but left any earlier non-rejection banner (e.g. a
failed image upload) in place. maybeReconnectOnPageResume refuses to
reconnect while a banner sits on a closed PTY and the reconnect overlay
hides behind a banner too, so the tab came back disconnected with no way
to recover. Clear the banner in the deferral branch like the normal
reconnect path does.
Test: hidden-tab 1001 close after a failed upload now opens a second
socket on visibilitychange (1 -> 2), red before the fix.
Follow-up to the salvaged #111918. That commit fixed the WebGL context
pile-up by removing the WebGL renderer altogether. With @xterm/xterm 6 the
fallback is the DOM renderer, so wide layouts would have lost the crisp,
fast rendering the renderer split in 63975aa deliberately gave them, and
the salvaged comment ("default canvas renderer") described a renderer that
no longer exists.
The leak itself is one missing call: @xterm/addon-webgl 0.19 removes its
canvas on dispose but never calls WEBGL_lose_context.loseContext(), so
every PTY reconnect (which rebuilds the Terminal) leaves a live GL context
until GC. Browsers cap live contexts at ~16 and force-lose the oldest, so a
reconnect storm eventually blanks the terminal the user is looking at.
`loseWebglContexts(host)` runs in the terminal effect's cleanup before
`term.dispose()` and releases every WebGL context under the host; WebGL
stays on for wide layouts exactly as before.
The keepalive is no longer gated on tab visibility / chat activity: an open
PTY socket on a hidden or backgrounded tab still owns its PTY and is the one
most likely to sit idle through a proxy timeout, and the frame is ~20 bytes.
That removes the `shouldSendPtyKeepalive` helper and its two tests; the
socket-open check already lives in `sendTerminalResize`. The ChatPage test
drops its "no WebglAddon constructed" assertion, which passed on base too
(jsdom hosts are 0px wide, so the wide-layout WebGL branch never ran).
Fold the four salvaged cases into one predicate truth table (WSL drive
mounts and .cmd shims refused, /mnt/data and /usr/bin accepted) and one
resolver test that walks the real re-scan: PATH interop hands back a
Windows npm first, the scan skips the /mnt/c entry and accepts the native
npm under /mnt/data. Drops the dead `hermes_cli.main._is_windows` patch —
the resolver calls main_install_repair's own `_is_windows`, which is
already False on the POSIX hosts these tests run on.
reap_terminal_workers signalled any retained worker on the first tick after
its run closed, but a healthy worker is still alive for a moment after
kanban_complete / kanban_request_review returns (final assistant turn,
session persistence), so slow-but-healthy workers were killed mid-finalisation
and logged as terminal_worker_reaped. Reap only runs whose ended_at is at
least TERMINAL_WORKER_REAP_GRACE_SECONDS (120 s, two default ticks) old;
the fingerprint check is unchanged. Each row is now handled on its own so a
signal or /proc failure on one run is logged and skips only that run.
Tests: a just-closed run is not signalled and keeps its evidence, then is
reaped once the grace has passed (red before); one raising row no longer
aborts the sweep for the others (red before).
A worker that called kanban_complete and then hung (e.g. holding deleted
state.db-wal/-shm inodes, which trips the DeletedWalGenerationError guard on
every later write) was unreachable by any command: the terminal transition
cleared tasks.worker_pid, _end_run cleared task_runs.worker_pid too, and every
reclaim sweep only looks at status='running' cards (#111791).
Keep the evidence and add the consumer: task_runs gains worker_started_at (the
spawn-time fingerprint tasks already carry), _set_worker_pid stamps it, and
_end_run leaves worker_pid / worker_started_at / claim_lock on the closed row.
reap_terminal_workers runs in the dispatcher's reclaim phase (every tick and
`hermes kanban dispatch --once`): a host-local pid on a closed run that is
still the fingerprinted process is terminated through the existing
_terminate_reclaimed_worker (SIGTERM, then SIGKILL after the poll window) and
recorded as a terminal_worker_reaped event; a pid that is gone or recycled
only has its evidence cleared; legacy rows without a fingerprint are never
signalled.
Slimmer redo of PR #111798 by @KoNit-K: same schema + retention shape, but the
reaper reuses _worker_alive / _terminate_reclaimed_worker(started_at=) instead
of a second start-time reader and a guarded-kill closure, scans every closed
run instead of a task-status allowlist, and clears dead evidence so rows are
not rescanned forever.
Fixes#111791
Only Task.from_row coerced BLOB-typed cells; a BLOB task_comments.body (or
event payload / run summary) still came back as bytes and crashed
`hermes kanban show <id> --json` with "Object of type bytes is not JSON
serializable". Apply _lossy_text in the other from_row constructors.
Test: BLOB comment body and event payload -> str with U+FFFD and
JSON-serialisable (red before).
A tasks row whose TEXT body holds invalid UTF-8 made sqlite3 raise
"Could not decode to UTF-8 column 'body'" inside fetchall, so `hermes kanban
list` (and `show`, and every other reader of that row) failed board-wide
until the row was deleted by hand; a BLOB-typed body came back as bytes and
crashed `--json` (#111743).
Fix it once at the connection: every board connection (`_open_configured`
and the read-only descendant path in `connect`) installs a lossy
text_factory that substitutes U+FFFD, and `Task.from_row` runs BLOB cells
through the same helper so a corrupt row renders with replacement
characters instead of taking its neighbours down.
Fixes#111743
The PR docstring said complete_task applies "the same fence request_review
applies", but the two disagreed: complete_task keyed on a live worker
process while request_review still refused any running task with a
claim_lock, so a claim whose worker is gone (or a CLI/library claim that
never spawned one) could be completed but not sent to review without
force. Factor the liveness test into _claim_is_live and use it in both.
TTL expiry is deliberately not part of "live": reclaim_stale_tasks extends
(not reclaims) the claim of a live worker, so the process stays the
liveness authority.
Test: claim -> request_review without a worker PID now succeeds (red
before); a live worker's claim is still refused without expected_run_id.
The first cut refused every claim-less complete of a running+claimed card,
which also refused the flows that have no worker to protect: a library or
CLI claim that never spawned a worker, and a worker whose process is gone
(12 sibling tests exercise exactly that shape). The guard now fires only
when tasks.worker_pid names a process that is still alive under its spawn
fingerprint (_worker_alive), which is the run the issue asked us to keep
open. Test updated to stand in as the live worker via _set_worker_pid.
complete_task authorised a terminal transition by task status alone; the
`current_run_id = ?` fence only applied when the caller volunteered
expected_run_id (derived from HERMES_KANBAN_* env). A human at the CLI, an
orchestrator session or any env-less caller therefore marked a `running`
card done and _end_run closed the dispatcher worker's run row while that
worker kept executing (#111764).
Mirror the fence request_review already carries: a `running` task under a
live claim needs expected_run_id (worker ownership) or force=True (explicit
operator override), otherwise LiveClaimError. `hermes kanban complete
--force` and the dashboard's "mark done" (a human action) carry the override;
the kanban_complete tool reports a structured refusal. Completing `ready`,
`blocked` or `review` cards without a claim is unchanged, so the manual /
orchestrator flows PR #73188 pinned keep working.
Fixes#111764
`hermes kanban dispatch` (plain output), the standalone daemon's stuck warning
and the gateway's embedded dispatcher stuck warning all reported a bare
`Spawned: 0` / "0 workers spawned" while the respawn guard held every ready
card — the reason existed only as a `respawn_guarded` task event visible via
`hermes kanban tail`. Operators watching the gateway health warning for 73+
ticks (#111910) had nothing to act on.
- `kanban_db_dispatch.describe_suppression()` renders the guard reasons per
task plus rate_limited / skipped_locked / memory_pressure for one or more
DispatchResults, so the CLI daemon and gateway warnings share one wording:
`Last tick held back: active_pr=1, memory_pressure=elevated.`
- plain `dispatch` output prints `Guarded (<reason>): <task id>` and the
tick-level holds, mirroring the JSON fields.
- kanban docs: how to see why a ready card is not spawning.
Co-authored-by: Steven Saehrig <trac3r726@users.noreply.github.com>
Part of #111910
`hermes kanban dispatch --json` only emitted `spawned` and the skip buckets it
already knew about, so a ready card held by the respawn guard (`active_pr`,
`recent_success`, ...), a quota-released worker, a lost dispatch lock or a
memory-pressure hold all looked like `spawned: []` with no reason. Emit
`respawn_guarded`, `rate_limited`, `skipped_locked` and `memory_pressure`
from the DispatchResult the tick already returns.
Salvaged from #111917 by @KoNit-K. Dropped hunk: the `_ACTIVE_PR_RECOVERY_LANES
= frozenset({"review"})` rename in kanban_db_dispatch.py, which is behaviour-
identical to the existing `lane == "review"` check and does not implement the
role-aware exemption the issue asks for.
Part of #111910
The dashboard's estimate endpoints make the same headless auxiliary call
as specify/decompose but never bound an affinity scope, so they still sent
no x-opencode-session and the OpenCode Go relay answered 400
MissingSessionID (#112043). Declare kanban:<task_id> for an existing task
and a stable kanban:estimate key for the create dialog (no task yet),
unless a scope is already bound.
Test: _run_estimate captured header None before; now kanban:t_1 /
kanban:estimate and nothing leaks past the call.
`hermes kanban specify|decompose`, the dashboard specify/decompose routes and
the gateway auto-decomposer all reach the LLM through
hermes_cli/kanban_specify.py::_call_aux outside any agent turn. No
conversation affinity scope is bound there, so agent/opencode_affinity.py
resolved an empty key and sent no `x-opencode-session`; the OpenCode Go relay
rejects such requests with 400 MissingSessionID and the user sees
"Specify failed: LLM error: BadRequestError".
Declare a per-task affinity scope (`kanban:<task_id>`) around the call —
the same host-declared scope the main turn, compression and the
OpenRouter/Portal sticky keys already resolve first — but only when no scope
is bound, so an in-turn caller keeps its conversation's key. Reset in a
finally so nothing leaks past the call.
Live: real httpx transport capture against auxiliary.triage_specifier
provider=opencode-go — before: no x-opencode-session header; after:
`kanban:t_45567533` on specify and decompose, stable per task, distinct per
task; a pre-declared scope is preserved; an openai route gets no header.
Fixes#112043
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Follow-up to the salvaged assertion: the elapsed-grace contract is what
the implementation promises, but the cancellation must still come from
the renewal-error path (>= 1 failed renewal observed), or a stub that
never gets called would pass. Docstring names why a minimum attempt
count was the flake (#111471).
_resolve_cron_enabled_toolsets returned None when _get_platform_tools
raised, and AIAgent reads None as "load every toolset": a malformed
platform_toolsets block (or a stale-module import error after an update)
turned the operator's cron restriction into the full default set, with
only a log warning. Unattended jobs process untrusted text, so that is a
privilege widening, not a safety net (#111380).
The resolver now raises a RuntimeError naming the cause; run_job's
existing failure path records it on the job (last_error, failure streak,
incident) and the agent is never constructed. Per-job enabled_toolsets
(unknown names included) and the MCP merge path never touch the
platform resolver and are unchanged; the disabled-toolset resolver has no
fail-open branch.
Live: platform_toolsets: oops -> before: run ok, enabled_toolsets=None,
98 tool names selected; after: run fails "Cron toolset resolution
failed, so this run was refused rather than given every tool", agent
never constructed. normal / unknown-per-job / mcp-merge shapes: identical
before and after.
Co-authored-by: Austin Bell <10687162+robertaustinbell@users.noreply.github.com>
The completed-occurrence dedup gate (due scan in _evaluate_due_job and the
fire claim in claim_job_for_fire) consumes the due slot and advances
next_run_at without a run and without a ledger row, so before this change
a skip left zero trace: no log line, no execution, last_status untouched
(#111414 reported exactly that silhouette). The gate now logs a WARNING
naming the job, the skipped instant and the completed execution that
already covers it, from the one seam both gates share.
The reporter's root mechanism — an off-tick run stamping the NEXT
occurrence's identity onto its row, which the dedup later honoured — is
already closed on main (ac10770894, a73b750391, cd685a22e6; 82ae78cc11
ignores such backdated rows), so a stale-stamped row fires normally; only
a genuinely completed slot is skipped, and now says so.
Co-authored-by: holny <holny@foxmail.com>
Follow-up to the salvaged #111528 hunk: _prepare_job_prompt passed the monitor
block through a conditional kwargs dict although _build_job_prompt already
treats a falsy runtime_data_prompt as absent. Drop the duplicate unit-level
positive test — the run_job-level test in test_monitor_kind.py covers the
same invariant through the real path — and keep the strict-user-prompt
control.
`hermes update` run while the Desktop app is open ended `partial`/exit 1 and re-armed
`fleet_restart_pending` on every run: `_gateway_recovery_partition` exempts a
`supervisor == "desktop"` serve from restart (`_DESKTOP_SERVE_SKIP_REASON` — it hosts the
live Desktop chats), while `match_runtime_outcomes` counted that same still-alive process
as `unaccounted` whenever the survivor probe found its pre-update incarnation in the ledger.
Nothing in the updater is allowed to discharge that obligation, so it could never finalize.
- update_inventory.match_runtime_outcomes: a Desktop-supervised serve/dashboard still alive
reconciles as a new outcome `deferred` (handed back to its supervisor). "restarted" would
be untrue — the process provably runs old code and the Desktop app does not respawn it after
a terminal-side update. A gone one stays `restarted`; a manual/systemd survivor stays
`unaccounted`.
- update_inventory.report_unaccounted_runtimes: prints `deferred` rows with the one remedy
that exists (relaunch the Desktop app) without escalating; the `systemctl --user restart
hermes-serve.service` hint is Linux-only now (it was shown on macOS too).
- update_abort_recovery: same class on the fresh-child recovery path — `_owed_stale_serve_rows`
excludes Desktop-owned survivors from `_abort_recovery_is_complete` and the incomplete
gate in update_cmd_fleet; they are still named by `_warn_stale_serve_runtimes` and kept
in the receipt's `stale_runtimes`.
- tests: end-to-end (exit 0, receipt `success`, `runtime_outcomes` gateway=restarted /
serve=deferred, marker cleared, relaunch hint printed) + abort-path predicate; both red on
origin/main.
Fixes#111494
Supersedes #111499
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Reshape of the cherry-picked fix from #104194:
- The SOUL.md gate + `register_profile_gateway(start_now=False)` now live in
`hermes_cli/service_manager.py::register_unregistered_profile_gateway`, next to the s6
manager and `_profile_dir_for_gateway_service` it needs, instead of a private reach-in
from the 6.5k-line `hermes_cli/gateway.py` facade. The facade only decides "start
repairs, stop/restart re-raise" and keeps ONE error handler (S6Error is a RuntimeError;
register's ValueError/RuntimeError/OSError surface as the same `✗` + exit 1).
- Tests trimmed from four to two invariants: start on a real profile registers `down`
and then starts; stop on an unregistered profile / start on a directory without
SOUL.md keep the original error and mint nothing (parametrized). Dropped: the
registration-failure traceback test (covered by the single except clause) and the
duplicate no-marker/stop split. The test now resolves the profile dir through the real
HERMES_HOME mapping instead of monkeypatching `_profile_dir_for_gateway_service`.
- Docs: docker.md multi-profile section says `gateway start` inside the container
registers a slot for a profile created from the host.
`profile create` registers an s6 gateway slot only when the creating process is
itself inside the container: `detect_service_manager()` reads `/proc/1/comm` in
the CALLER's PID namespace, so on a host whose `~/.hermes` is bind-mounted into
the container the hook is a silent no-op. The profile directory lands exactly where
the container reads it, but no slot exists, and `hermes -p <name> gateway start`
inside the container fails with "not registered" until the operator restarts the
container so the boot reconciler notices. Same symptom as #54174 reaches `profile
install` by the same route.
Register the slot on demand instead. When `start` hits `GatewayNotRegisteredError`
and the profile directory carries `SOUL.md` — the boot reconciler's own "real
profile" marker — create the slot and start it. That makes
`_maybe_register_gateway_service`'s promise true without a restart.
Deliberately narrow:
- Only `start` self-heals. `stop` and `restart` on an unregistered profile keep the
original error; registering a slot in order to stop it would be absurd.
- `SOUL.md` gates it, so a mistyped `-p` name or a stray directory cannot mint a
phantom slot for a profile that does not exist.
- Registers with `start_now=False` and then goes through the ordinary `start` path,
so the `desired_state` write that lets boot reconciliation restore want-up after
a container restart keeps a single owner.
- `ValueError` (slot appeared underneath us) and `RuntimeError` (s6-svscanctl
failed) surface as the existing actionable error, never a traceback.
Refs #54174. PR #54182 fixes the narrower `profile install` case by adding the same
registration call; this closes the symptom for any existing profile directory and
without a container restart.
gnome-shell moves a launched ShellApp from STARTING to STOPPED when the
startup-notification sequence completes or times out (mutter, ~15 s), not
when the process exits. finish() healing right after an exit-without-reveal
(boot crash, early quit) therefore wrote the entry during STARTING — the
exact #111906 arming condition. Only the reveal byte from Electron heals
now; the wake byte finish() writes just unblocks the reader. A skipped heal
is picked up by the next terminal/updater launch or revealed grid launch.
- cmd_gui with DESKTOP_STARTUP_ID: the entry is installed only after the fake Electron
writes the reveal byte to HERMES_DESKTOP_READY_FD (spawn precedes install).
- cmd_gui without DESKTOP_STARTUP_ID (control): install still precedes the spawn.
- DeferredDesktopEntryInstall.finish(): an app that exits without revealing heals
exactly once, after the exit.
- linux-launcher-ready: one byte, fd closed, variable consumed; garbage/absent = no-op.
Docs: user-guide/desktop.md explains when the entry write happens for grid launches.
`hermes desktop` used to create/refresh `~/.local/share/applications/hermes.desktop`
synchronously before spawning Electron. When the entry is ABSENT (first run after an
update, deleted by a cleaner, tombstoned by AV) that write lands while gnome-shell still
has the grid-launched ShellApp in STARTING; unpatched shells (before GNOME MR !4428)
drop the app's last strong reference on `installed_changed` and the next idle GC kills
the whole Wayland session, minutes to an hour later (#111906, residual after #111396).
Now a launch that carries `DESKTOP_STARTUP_ID` (app grid / menu) defers the write:
the launcher opens a pipe, hands Electron its write end as `HERMES_DESKTOP_READY_FD`
(pass_fds), and a worker thread installs the entry once Electron reports the main
window revealed (`onRevealed` in createWindow, via the new
`apps/desktop/electron/linux-launcher-ready.ts`), plus a 2 s settle so the compositor
has mapped the surface. If Electron exits without ever revealing a window, `finish()`
installs the entry after the exit (STOPPED app, no STARTING object) — self-heal
semantics survive. Terminal launches, the updater's detached relaunch and
`--build-only` have no DESKTOP_STARTUP_ID / spawn no app and keep writing immediately.
Why not simply write after Electron exits (#111915's approach): a daemon thread
started right before `sys.exit` is killed with the interpreter, so the heal is lost,
and a heal that waits for the user to quit the app leaves the menu entry missing for
the whole session.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Turn the SIGTERM→SIGKILL deadline in `_kill_pids_posix` into `_POSIX_TERM_GRACE_SECONDS`
(10.0s) documented against what it has to outlast: `web_server.py::_lifespan` blocks on
`stop_hosted_room_service(timeout=5.0)` + `join(1.0)` before `PTY_REGISTRY.close_all()`
(≤1.5s per attached Chat PTY). The 3.0s deadline predates the hosted-room stop (added
2026-08-30) and SIGKILLed the backend mid-teardown, so its ui-tui / tui_gateway.entry
children were never closed and kept the deleted state.db-wal inode open — the next
`hermes` start refused with FATAL DeletedWalGenerationError (#111912).
Two invariant tests against real child processes: a teardown as long as the lifespan
budget finishes gracefully (red on the old 3.0s deadline); a SIGTERM-ignoring process is
still SIGKILLed at the deadline. The orphan reaper's 1.5s grace is left alone on purpose
— it runs on the Desktop boot path under a 10s ready-probe — and says so in the comment.
Co-authored-by: Tranquil-Flow <66773372+Tranquil-Flow@users.noreply.github.com>
Only names in OPTIONAL_ENV_VARS / _EXTRA_ENV_KEYS and the platform *_HOME_CHANNEL /
*_ALLOWED_USERS suffix family route to .env; other documented ALL-CAPS names still land in
config.yaml as top-level scalars with a notice. Say so instead of 'every documented
environment variable' (#111848 stays open for the remaining names).
compression.model_thresholds.<model>, terminal.docker_env.<VAR>, lsp.servers.<lang>.*,
auxiliary.<task>.extra_body.<k> and similar free-form mappings are declared as {} in
DEFAULT_CONFIG; the fail-closed gate walked into the empty dict, found the user-chosen key
missing and refused the write. An empty dict now accepts the rest of the path, like a scalar
leaf or a platforms container does. Populated sections keep the did-you-mean refusal.
`display.tool_progress_overrides` (deprecated) and the fabricated `agent.max_iterations`
were unknown sub-keys under known sections, the shape `hermes config set` now refuses
before writing. The tests exercise value-shape parsing and section creation, not those
keys, so they use `mcp_servers.demo.env` and `agent.max_turns` instead.
`hermes config set gateway.discord.gateway_restart_notification true` wrote the
typo into config.yaml and only then printed the "not a recognized config key — it
was saved anyway" notice (#112003). Under a KNOWN section an unknown sub-key can
only be a typo, so `set_config_value` now exits non-zero via `_exit_invalid`
before reading or writing config.yaml, with the did-you-mean hint.
Scope preserved from ed3a0b3 (warn-after-write): unknown TOP-LEVEL keys are
still written with the post-write notice, because top-level scalars are bridged
into os.environ for skills/external apps and that namespace is open by design;
the `_OPEN_SUBKEY_TOP_LEVEL_KEYS` / platform-container exemptions in
`_validate_config_key` are untouched, and `--force` keeps writing anything. This
is the fail-fast piece the maintainer scoped in the close comment on #111133.
`_validate_config_key` also suggests the path minus its wrong prefix
(`gateway.discord.x` -> `discord.x`) when no same-level sibling is close; the
headline typo previously produced no hint at all.
Docs: cli-commands.md `set`/`unset` rows, configuration.md tip, `--force` help.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
`hermes config set FEISHU_HOME_CHANNEL oc_x` wrote the top level of config.yaml
while the platform setup flows and /sethome write the same name to .env via
save_env_value, so two writers fed two readers: the gateway bridges the yaml copy
into the environment only when .env lacks the name, one-shot CLI readers never
bridge, and the two copies diverged silently (#111848). Only credential-shaped
names were routed to .env because `_is_env_config_key` is the provider-credential
predicate.
Follow-up to KoNit-K's cherry-picked fix (#111850), which routed the
`setup_hidden_env` suffix family: the predicate now lives in the topical sibling
`hermes_cli/config_env_routing.py` and covers every bare name Hermes itself
registers as an environment variable (OPTIONAL_ENV_VARS, _EXTRA_ENV_KEYS — "env
var names written to .env" — plus the setup-hidden suffixes for plugin adapters
nobody enumerated), so `*_ALLOWED_USERS`, `WHATSAPP_MODE`, `MATRIX_PASSWORD` and
the rest of the adapter-saved family take the same file. `set` and `unset` also
drop a stale same-named top-level config.yaml copy so the reporter's drift cannot
come back, and `get` resolves .env first then that copy — the gateway's own read
order. Provider credentials keep the credential_lifecycle rotation path.
Docs: environment-variables.md tip, hermes_cli/AGENTS.md config rule.
`launchctl kickstart system/<label>` needs root. When a LaunchDaemon-supervised
backend does not come back and `hermes update` runs as a regular user, the
manual hint now reads `sudo launchctl kickstart -k system/<label>`, so the
operator gets a command that works from the shell they are in. LaunchAgent
(gui/user domain) hints are unchanged.
`hermes dashboard --stop` decided its exit code by re-scanning the process
table after the kill. On macOS a launchd KeepAlive job brings the backend
back on a fresh PID within the grace window, so the re-scan found it and
--stop exited 1 right after printing that the stop succeeded and that the job
restarts itself. Judge the exit code from the kill result instead: exit 1
only when a matched pid could not be stopped.
_launchd_job_owning_backend matched a process to a job only on exact argv
equality with the plist ProgramArguments, but _respawn_dashboard_processes
appends --no-open to dashboard commands. A detached copy created by an
earlier update therefore never matched, was treated as a manual backend and
was respawned detached again on every update. Ignore --no-open on both sides
of the comparison.
`hermes doctor` printed the same informational "WAL journal mode" line for a database
that is still WAL although the operator configured `database.journal_mode: delete`. The
runtime never live-downgrades an existing WAL database (a downgrade under open
connections corrupts it) and says so only once per process in the gateway log, so the
one surface operators check told them they were protected when every process was still
writing WAL on the filesystem they configured `delete` for (#111729).
Doctor now compares the configured mode (`resolve_journal_mode`) with the on-disk header
and warns `<db> is in WAL mode despite database.journal_mode=delete`, naming the
never-live-downgraded rule and the one-time offline `PRAGMA journal_mode=DELETE`
remedy; a vulnerable SQLite still counts the database as WAL-reset exposed. This check
outranks the cross-VM hint (whose remedy, "set journal_mode: delete", is already
applied). A configured `wal` is unchanged.
Core hunk ported from #104714 (@jonpol01); its holder enumeration
(`foreign_state_db_holders(include_scan_gaps=True)` + per-PID report) is not included
here — that half is a 600-line change under separate review.
Keep one kill-path test for the reporter's scenario (a detached copy of a loaded
launchd job is kickstarted through launchd, never respawned; a failed kickstart is
unrecovered with the manual command; an unrelated loaded job leaves the manual
respawn intact) and one host-independent pure-function test for the plist scan and
PID attribution (platform forced to darwin so it runs on Linux CI too, instead of
the macos_only marker that would have skipped it here).
The other nine tests from #111690 asserted the same paths per branch; the salvage
bar is 1-2 invariant tests proven red on base, and both of these error on
origin/main (no _loaded_launchd_backend_jobs / _launchd_job_owning_backend).
Adopted from the sibling PRs in this cluster, with thanks:
- #99807 (@Chevron7Locked) and #101541 (@ivcdigital26): a kickstart that returns 0 is only
"restart requested" — success now requires launchd to report a live PID other than the one
that was stopped (the gateway's _wait_for_launchd_service_pid, 15 s), and a job that never
comes back is reported with the manual `launchctl kickstart -k` command instead of a tick.
- #89793 (@mettamyron) and #101541: a plist may wrap the backend in `/bin/sh -c …` without
exec, so launchd's live PID is the shell and the stale backend is its child — ownership now
follows the parent chain (bounded, cycle-safe).
- #89793: `hermes dashboard --stop` on a launchd-owned backend now says a KeepAlive job
restarts itself and gives the `launchctl bootout` command, instead of the misleading
"restart it yourself" hint. The job scan runs for --stop too, on macOS only.
`hermes update` snapshots each stale dashboard/serve backend's systemd unit before the
kill and respawns everything else from its captured argv (#40449, #68934, #78821). On
macOS that "everything else" includes a backend supervised by a launchd job: the
respawn comes back detached and sits on the job's port, launchd's KeepAlive then fails
every restart of the job with "port already in use" (exit 75, one attempt per ~10 s,
forever), and the backend that IS serving is no longer supervised. Every later update
kills the copy and respawns it again, so the loop outlives the update that started it.
The update now snapshots the loaded launchd backend jobs before the kill (LaunchAgents
in the gui/user domains, LaunchDaemons in system; only plists whose ProgramArguments
are a dashboard/serve backend are probed), attributes a PID to a job when launchd
reports it as the job's live process or the process runs the job's exact
ProgramArguments (the detached copy an earlier respawn left behind), and brings such a
PID back with `launchctl kickstart <domain>/<label>` instead of an argv respawn. A
failed kickstart is reported with the manual command and counted as unrecovered, like
a failed systemctl restart. Linux and Windows paths are unchanged.
teknium1