Best-effort bootout calls (unloaded-job recovery, stale-EIO retry,
plist refresh, uninstall) and the handled kickstart -k in
launchd_restart inherited the terminal's stderr, so an expected
unloaded job printed raw launchctl errors around the CLI's own lines:
Could not find service "ai.hermes.gateway" in domain for user gui: 501
↻ launchd job was unloaded; reloading
Boot-out failed: 3: No such process
Capture them with _CAPTURE_TEXT instead. The kickstart error stays
available as e.stderr for the update_cmd failure diagnostic, and the
post-bootstrap kickstart intentionally stays loud (its failure feeds
the domain-unsupported fallback). Same precedent as the reload
helper, which already runs bootout with 2>/dev/null.
[salvage: picked hermes_cli/gateway.py only; the two capture_output kwarg-assertion tests are
replaced by fd-level invariant tests with a fake launchctl in the follow-up commit]
Keep the loop-breaker test (bot quote-reply processed on main, dropped with the
flag, explicit @mention still passes) and the human quote-reply test (unaffected
with the flag on). Drop the defaults-off snapshot, the /cmd@botname variant, the
plain-chatter duplicate and the own-echo helper test: they re-cover paths
(_message_mentions_bot, _is_own_message) already pinned elsewhere in this file.
An aux task pinned to a private endpoint via `base_url` (a home Ollama
box at `byron.local`, a LAN IP, localhost) is the intended per-task
endpoint feature and can never bill a provider. The Settings → Model
banner still counted it as "still run on openai" forever and offered
"Reset all to main", which would wipe the working local setup; the
post-switch `stale_aux` report had the same blind spot; and the aux row
never showed the `base_url` the backend already sends, so the pin was
indistinguishable from a paid-provider pin.
- `GET /api/model/auxiliary` now stamps each task with `local_endpoint`,
the verdict of the one canonical classifier
(`agent/model_metadata.py::is_local_endpoint`) — no TS mirror of the
private-range rules, so frontend and runtime cannot drift.
- Desktop: the persistent banner filter is the pure
`staleAuxAssignments()` and skips `local_endpoint` pins; the pinned row
appends ` · <base_url>` when one is set.
- `_stale_aux_pins` (post-switch report) skips local pins the same way.
- `is_local_endpoint`: `*.local` (RFC 6762 mDNS) now counts as local, and
IPv6 literals no longer ride the "no dots ⇒ unqualified host" rule, so
a global-scope address (`2607:f8b0::1`) is not local while `::1`,
ULA and link-local still are via the `ipaddress` scope checks.
Slim redo of #106236 (@webtecnica) and #106234 (@huklaa), which fixed the
same symptom with a client-side classifier copy; the bug class, row
display and mDNS/IPv6 classifier corrections are theirs.
Refs #106228
Co-authored-by: Hukla <129692708+huklaa@users.noreply.github.com>
Co-authored-by: webtecnica <webtecnica@gmail.com>
Move the WSL / /dev/dxg / d3d12_dri.so probes into _prefer_wsl_d3d12 with
the probed paths as module constants, so the launcher call site is a single
line and a test can lay out a fake WSLg host without touching real
/dev or /usr/lib. The two tests now run the real _desktop_launch_env end to
end (selected under WSL+dxg+driver; untouched with an explicit Mesa
override, off WSL, without /dev/dxg, or without the driver file) instead of
unit-testing the helper with a precomputed boolean.
Docs: one paragraph in the Desktop guide on the automatic selection and
the env vars that keep an explicit choice authoritative.
Follow-up to Xipong's fix for #106117 (salvaged from #106118).
Fold the helper into _npm_lifecycle_env itself: the whole fix is one
is_file() check plus a setdefault, so a separate function, the
try/except around get_hermes_home() (it never raises) and the
dict-returning indirection were shape-gate violations. Build the path
with os.fspath so it is correct on Windows too.
Tests: keep the two invariants (file present -> NPM_CONFIG_USERCONFIG
points at it; explicit process/caller value wins and a missing file
sets nothing), fold the other two into the negative test.
Docs: one paragraph in the desktop troubleshooting page next to
ELECTRON_MIRROR describing $HERMES_HOME/npmrc.
Refs #106373
- _ssl_interop_hint: also match ssl.SSLError instances (and one level of
__cause__/__context__) plus the bare UNEXPECTED_EOF marker, so an
SSLEOFError whose text httpx did not repeat still gets the hint. The
hint now names the TLS 1.2 diagnostic and links the providers docs
note instead of an issue number.
- tests: 3 -> 2 invariants (parametrized login_post/poll SSL case keeps
the raw text + hint + cause; a plain httpx timeout gets no hint).
- docs: providers.md Codex note carries the reporter's exact openssl.cnf
classic-groups snippet (EN + existing zh-Hans copy).
Refs #106384. The TLS max-version cap itself stays PR #44392's scope.
Device-login requests on networks whose middlebox rejects the larger
TLS 1.3 ClientHello sent by OpenSSL 3.5+ (post-quantum hybrid groups)
fail with SSLEOFError / handshake timeouts while curl still works, so
they masquerade as a Codex outage (#106384). The polling loop let the
raw httpx error escape unshaped, and _codex_login_post dropped the
exception chain and gave no actionable hint.
- add _ssl_interop_hint() applied to both device-login transport paths
- re-raise _codex_login_post failures with 'from exc' to preserve cause
- wrap the poll POST so transport failures become a shaped AuthError
(device_code_poll_error) carrying the SSL detail and OPENSSL_CONF
workaround hint; KeyboardInterrupt handling is unchanged
(cherry picked from commit 8cd94c36ce8437db5b00290b9edbedcd2116c02c)
Six new tests collapsed into three that each go red without the indexed
projection: bounded VM steps for a page read and for an append (the
symptom), composite user-handoff identity + first-row position + internal
columns never leaking + signed-zero timestamp identity (the parity
contract), and a legacy store read-only then lazily migrated (the
persistence contract). The parametrized symmetry/visibility-toggle test
exercised the same key through raw SQL edits no production path performs
and passed on base, so it was a change-detector for the trigger shape.
The salvaged fix added `_pid_record_matches_home`, a near-copy of
`recorded_gateway_home_conflicts(record, expected_home=...)` (already the
scoped-home predicate used by the #89315 stop guard). Reuse it instead of
carrying a second helper with the same semantics (legacy records without
`hermes_home` are accepted by both).
Also trims the salvaged tests to the two invariants that prove the symptom
(scoped probe reports a live foreign profile's PID and leaves its
identity files intact; a dead-PID scoped record is still cleaned). The
third-home-claim variant exercised the same `saw_live_pid` branch.
The live-record unlink protection must only apply to scoped reads
(polling another profile must not delete its identity files). The
unscoped path keeps main's behavior: a live record owned by another
home inside this home's gateway.pid is unlinked on refusal, per the
#89315 cross-profile stop contract
(test_cross_profile_kill_refusal.py).
(cherry picked from commit 1c7343f9c0175867b9649ab8c54cf924d04e2266)
get_running_pid(pid_path) validated records against the serve process's
HERMES_HOME, so a dashboard scoped status poll (?profile=) rejected a live
foreign profile's record, force-unlinked its gateway.pid/gateway.lock, and
reported running=false. Validate scoped queries against the probed home
(pid_path.parent, like the expected_home sibling) and never cleanup-unlink
a live record's identity files; dead-PID records still clean up.
(cherry picked from commit 472ca1b8d87df5246d83864e9b1492d06e99e8e5)
When the redirect cap trips, the correction that cancelled the final attempt is
still sitting in _pending_redirect; finalize_turn's clear_interrupt() would drop
it silently. Drain it into the steer slot so it rides result["pending_steer"] and
becomes the next user turn on every surface that already honours that key.
Both new exit reasons get a turn-completion explanation so the user sees why the
turn stopped instead of an empty reply.
Collapse the four class-based tests into two parametrized invariants over both
refunding restart flags and move them to tests/agent/ (the phase modules live in
agent/): a single restart still refunds-and-continues; a re-armed restart breaks
after max_retries refunds. The stub grows the redirect seam the follow-up commit
uses so the queued-correction contract is covered by the same test.
The redirect and rebuilt-for-fallback restart paths in apply_retry_restarts
refund the iteration budget and re-issue the iteration with no per-turn
bound. A redirect/interrupt that keeps re-arming the flag refunds forever,
so the turn loop never exits and the durable session turn lease is held
indefinitely (concurrent processes block up to LEASE_WAIT_SECONDS).
Add a per-turn restart_count accumulator (threaded through _run_phase like
the other loop locals) and break out once it exceeds max_retries, matching
the bound the compression path already has.
#106167 widened the base contract to SendResult but left six native-batch
overrides (Discord, Email, Matrix, Mattermost, Slack, Telegram) returning
None, which (a) meant a media-only reply on those platforms still reported
FAILURE because _record_delivery(None) records nothing, and (b) produced new
`ty` invalid-method-override diagnostics against the widened base (#106192).
Make the contract honest instead of annotating it Optional: each override
now rolls its batches (and any per-image fallback) into one SendResult, so
the turn-outcome accounting works on every platform, not just Signal and
the base loop. The "legacy overrides return None" comment in
_send_image_batch goes away with the legacy.
ty on the 8 touched files: origin/main 241 diagnostics / 20 override,
this branch 241 / 20 — byte-identical diagnostic set; the intermediate
`-> SendResult` head without this commit had 246 / 25.
Refs #106192
Thread record_delivery through _deliver_attachments, _deliver_media_attachments
and _send_image_batch so attachment sends feed the turn outcome tracker.
send_multiple_images (base default and Signal override) now returns SendResult
(success when at least one image/batch was accepted); _send_attachment_batch
returns bool. Legacy native-batch overrides that still return None record
nothing, keeping their previous behavior until migrated.
Fixes#106153
kanban_request_review now rejects reviewers that are not installed profiles
(#106163); the cross-surface lifecycle test used a bare "reviewer" name with no
profile behind it, which is exactly the phantom the guard exists to catch.
Use the module's `_check`/`_Reject` idiom instead of an inline
`return tool_error(...)` so every kanban_request_review validation
failure renders through the same path, and drop the unreachable
`or "none"` (list_profile_names() always contains "default").
Tests: compare the task's (status, assignee, run) tuple and the event
log before/after instead of the unordered 6-assert block, use the
context-managed kanban_db_connect.connect (the kb.connect alias is a
plugin-compat pointer — scripts/check_compat_pointers.py flagged it),
and reference #106163 in the invariant's docstring.
Salvage note vs #106214 (@gaoanze888): that PR guards the same condition
inside hermes_cli/kanban_db.py::request_review, but the DB primitive is
also the chokepoint for `hermes kanban request-review` and the
dashboard's drag-to-review, both operator surfaces where a non-profile
assignee (external/human review lane) is a documented board shape
(website/docs/user-guide/features/kanban-worker-lanes.md) — and it forced
five unrelated test fixtures to monkeypatch profile_exists to True. The
model-facing tool wrapper is the layer where a typo'd string is a bug,
so the guard lives there.
kanban_request_review(reviewer=<name>) reassigned the task to whatever
string the model supplied. A non-profile value (e.g. the literal
"reviewer") parked the card in `review` on an assignee the dispatcher
can never spawn, with no error to the worker — the chain stalled
silently (#106163). Validate the explicit reviewer against installed
profiles before touching the board and return a tool error listing the
installed profiles so the model can self-correct.
Salvage of #97429: the kanban_diagnostics `review_reopened` hunk and its
tests were dropped (main already replaced that loop with
`_latest_event_ts`; the tests targeted the PR's pre-refactor base).
Re-authored from the placeholder identity `regen <regen@local>` to the
PR author's GitHub noreply address (misconfigured local git, not malice).
Drop the sentinel-only batch test: a batch sentinel is already rejected by the
shared _is_clarify_non_response_sentinel list check that the existing sentinel
tests pin, so the case adds no new contract. Also add the contributor email
mapping for the cherry-picked commit so release CI can attribute it.
_sum_clarify only extracted the top-level ``user_response`` key, so batch clarify
results (questions=[...] -> responses[].user_response) fell through to the generic
placeholder and the summarizer never saw the user's answer/permission decision.
Closes#106077.
`hermes peer dm` resolves the target's canonical Bot Chat with
GET /api/sessions?title=Bot%20Chat&include_hidden=1. list_sessions_rich
admits the hidden root via the chain search, then _project_compression_tips
overwrites every surfaced field — title included — with the live tip's. The
title is carried root->tip by the agent AFTER publish_compression_child's
transaction; a rotation cut off in between (crash, closed app, the tip's
title write failing) leaves "Bot Chat" on the ended root and NULL on the tip,
so the projected row carries title=None, the handler's exact-title filter
drops it, the peer POSTs a duplicate and the UNIQUE(title) guard answers
400 "Title already in use" (#106165).
Fix at the projection: fall back to the root's title only when the tip has
none (a titled tip keeps winning). Same COALESCE in the bounded recent-
sessions lister, the other place that projects a lineage onto its tip. This
replaces the handler-level fallback in PR #106365 (a second lookup path
bolted onto _handle_list_sessions with try/except: pass) with a 5-line fix
at the one place the title is lost, so every list consumer sees the name.
Salvage of #106365 by @finn763.
Regression test from PR #106365 (one of its two tests kept: the full peer-dm
e2e; the listing-shape test asserted the same row and was dropped). The
handler-level fallback that shipped with the test is replaced by a SessionDB
fix in the follow-up commit, so this commit carries only the test.
Review follow-up for #106276: the compensating UPDATE can succeed and the
session row can still be retired before the verification read-back runs.
That window raised the same RuntimeError as a genuine restore mismatch.
Accept the absent session (warn + return) exactly like the pre-update
deletion window, keep strict verification for a surviving row, and add a
regression test that deletes the session after the UPDATE commits.
[salvage: test hunk dropped from this pick, see previous commit]
The exact-row rollback in restore_compression_failure_cooldown_row raised
RuntimeError when its UPDATE hit rowcount == 0, so a session row retired or
expired mid-attempt (e.g. by the maintenance sweep) crashed the turn
dispatcher during the compression summary failure path (#106271). A missing
session row means the cooldown died with it: nothing is left to restore, so
treat rowcount == 0 as a tolerated no-op (warn + return, skipping read-back
verification), mirroring the existing early-return for snapshots taken while
the session already did not exist. Write and verification failures still
propagate.
[salvage: contributor test file dropped from this pick; regression tests are carried from #106277 and trimmed to two invariants]
`hermes kanban promote --force <id>` printed `Promoted <id> -> ready` and
then the very next claim (a human `claim`, or the dispatcher tick seconds
later) demoted the task back to `todo` with `claim_rejected
{parents_not_done}` and returned None (#106195). The non-force refusal
even pointed operators at `--force` as the escape hatch.
The claim gate is deliberate: `claim_task` is the single enforcement point
("never ready -> running with an undone parent, whichever writer set
'ready'", cda20eec0c), and `complete_task`/`request_review` re-check the
same predicate, so a child let through by a forced claim could still never
finish. A promotion override therefore has no honest outcome; the
dependency edge is the real knob.
- drop `--force` from `promote` (parser, CLI handler, `promote_task`
kwarg, the `forced` event field nothing read)
- the refusal message now states why the gate cannot be bypassed and names
the working remedies: complete the parents or `hermes kanban unlink`
- two invariant tests: refusal on an undone parent leaves `todo` with no
fake `ready`; the flag no longer parses
Salvage direction from #75354 by @vyacheslavk (diagnosis of the promote ->
claim gap); the consume-at-claim authorization there is not taken because
the same parent gate also blocks completion of the forced child.
bridge.js carried its own copy of getMessageContent with the single-layer
envelope list, alongside an unused getContextInfo. With envelope peeling now
living in bridge_helpers.js, the copy would drift (a nested envelope around
a pollUpdateMessage is peeled by the helpers but not by the copy), so import
the shared one instead of keeping a second unwrapping list.
A quoted message can carry its own envelope stack (ephemeral
wrapping viewOnce wrapping the payload), and a reply itself can be
enveloped: both layers now unwrap iteratively (bounded) before quote
extraction. Peeled top-level envelopes return their inner message
immediately, as before — the template/buttons/list branches only
apply to unenveloped messages.
Keep the two tests that fail without the fix (one-shot -> recurring drops the
implicit times=1, recurring -> one-shot gains it) and fold the explicit-repeat
case into the first; drop the same-kind and explicit-finite cases, which only
pin behaviour the fix never touches.
`--in DIR` only chdir'd. Every cwd consumer (resolve_agent_cwd -> Codex
app-server thread cwd, the terminal tool, context-file discovery) prefers
TERMINAL_CWD over the process cwd, so a value inherited from a parent
Hermes surface, the shell or .env survived the chdir and the session kept
running in the old directory. The local backend was rescued by cli.py's
force-export at import time; docker/ssh backends and the TUI launch path,
which never imports cli.py, were not.
Refresh TERMINAL_CWD to the --in target when it is already set. An unset
variable stays unset so the backends keep deriving from the new process
cwd and no host path is pre-seeded into ssh/container backends.
Fixes#106220
Slim redo of the mechanism from #106410 on top of its pick (no wrappers, no
persisted "kind" enum, no process-local flag that dies with the process):
- Futility = the SAME holder PID set has blocked >= _FTS_HOLDER_FUTILE_ATTEMPTS
(10) deferrals over >= _FTS_HOLDER_FUTILE_SECONDS (30 min); tracked as
holders_since/holders_attempts in the persisted fts_rebuild_deferral record
and reset whenever the holder set changes. The 3-deferral/60 s escalate
window is the orphan-reap gate and stays as is.
- ONE escalated ERROR line names each holder pid + cmdline and the remedy that
can actually be followed from inside a gateway session: stop ONLY the other
holder; this process's own retry admits the rebuild within 60 s. The old
"with the gateway stopped" advice was unrunnable from a gateway-hosted
session (the gateway is the session) and is gone from both log and doctor.
- hermes doctor renders the futile record distinctly.
- retry_deferred_fts_recovery: a capped backoff earned by holder set X no
longer applies once the live holder set differs from X, so stopping the
other service is followed by a retry on the next tick, not up to an hour
later (the issue's 16-min wait).
- Tests trimmed from 5 to 2 invariants (futile line + doctor entry after N
same-holder deferrals; backoff reset when the holder set changes); the
contributor's control tests for changing PIDs / orphan reap are covered by
the existing test_repeated_deferrals_reap_inactive_orphan_then_rebuild.
The "canonical writes and LIKE search remain available" WARNING is kept
because it is true on origin/main: a stale open drops every FTS trigger, so
the messages INSERT succeeds (probed live with a real state.db + a second
process holding it). Writes fail only when a peer re-publishes triggers over
the corrupt index — a separate class, not this diagnostic.
Refs #106393
A supervised peer never satisfies the orphan reap, so stale-FTS repair retried forever with a misleading "canonical writes remain available" warning.
(cherry picked from commit e57f3a975d311aa44da1e92c5e727eba7c8cff70)
Tests that exercise the dashboard's gateway-restart path can end up
spawning a REAL `python -m hermes_cli.main gateway restart` child when
the spawn seam is not intercepted. `_spawn_hermes_action` launches it
with start_new_session=True, so it outlives the pytest worker; the
child inherits the pytest-tmp HERMES_HOME, which is not a profile and
hashes to no service suffix, so `get_service_name()` resolves the
DEVELOPER's `hermes-gateway` unit, `systemd_restart` restarts the live
gateway, and without systemd the fallback runs `run_gateway()`
in-process forever and squats the webhook port.
Live repro on this machine (origin/main): an unintercepted spawn of
["gateway", "restart"] from a test restarted the production gateway
(MainPID 136820 -> 1689090, NRestarts=1). On 2026-09-03 a sibling
refactor moved `_spawn_hermes_action` from the `hermes_cli.web_server`
facade to `hermes_cli.web_server_gateway` ~10 minutes before the tests
were repointed; runs in that window patched a name production never
read and left 39 orphans alive for six days.
The live-system guard now rejects any subprocess primitive whose
command line the canonical matcher (`gateway.status.
_gateway_command_subcommand`) classifies as `gateway run|start|restart`.
Argv substrings are never consulted, so `gateway status`, `gateway
--help`, `hermes_cli.main serve`, etc. pass through. Three files that
deliberately spawn and reap a stub child with a gateway-shaped argv
(flock holders, sleep sleepers with an argv tail) opt out with the new
`spawns_gateway_lookalike` marker, which lifts only this check and keeps
os.kill guarded. Two canary tests pin the block and the pass-through.
Defence at the exact boundary the incident crossed: systemd_uninstall() and
uninstall._remove_systemd_gateway() unlinked whatever get_systemd_unit_path()
returned. Before stop/disable/unlink, read the unit's own
Environment="HERMES_HOME=..." line (the parser status/refresh already use)
and, when it names a different home than this process, warn with both paths
and leave the unit alone. A unit without the line (hand-written) is still
removed as before.
With the previous commit a Docker/custom root (HERMES_HOME=/opt/data) gets a
hashed host-service suffix. Three callers used `_profile_suffix() or
"default"` as the PROFILE id, which is a different question: the s6
supervisor's slot for the root home is `gateway-default` regardless of where
the root lives, and the multiplexer's "am I a named profile" probe must not
treat a hash as a profile name. Route them through hermes_constants.
profile_name_for_home() (root -> "default", <root>/profiles/<name> -> name)
with the service suffix as the fallback for unknown layouts.
_profile_suffix() compared HERMES_HOME against get_default_hermes_root(),
which treats ANY home outside ~/.hermes (Docker /opt/data, a mktemp dir) as
"the root itself". Every such home therefore collapsed to the bare
`hermes-gateway` service name and the default profile's unit path
(~/.config/systemd/user/hermes-gateway.service); the documented
"else a short hash of the path" branch was unreachable.
A parity harness run with HERMES_HOME=$(mktemp -d) called
uninstall_gateway_service(), resolved to the production unit, ran
`systemctl --user stop/disable`, unlinked it and daemon-reloaded. With the
unit gone Restart= could not revive it: all cron jobs and every messaging
platform were down for 6.5 days.
Compare against the platform-native default home (~/.hermes) for the bare
name; keep the profile name for <root>/profiles/<name>; everything else
(temp dirs, Docker /opt/data) gets its sha256[:8] suffix as the docstring
always promised. The Docker image supervises with s6 (`gateway-<profile>`
slots), not systemd/launchd, so the bare host-service name was never load-
bearing there.