_teardown_session swallows agent.close() failures, so a "row stays open"
assertion alone is green even when close() aborted before
_finalize_owned_session_row. Assert the one side effect only that finalizer
produces (_owns_session_db cleared), give the stub the _memory_manager attr
close() reads directly, and assert the flag is untouched on the control
paths. Rename the sole-owner cross-process test to what it now asserts.
_finalize_session decides whether the TUI backend ends the state.db row
(gateway-owned source → no; another backend holds the lease → no; automatic
Desktop reclaim → no, #105588). _teardown_session then calls agent.close(),
whose _finalize_owned_session_row ends the same row as "agent_close" through
the agent's own handle whenever _end_session_on_close is still True — so every
spare decision was undone one call later. Verified with real SessionDB +
AIAgent.close(): with only the #105620 guard a Desktop ws_orphan_reap still
lands ended_at + end_reason="agent_close" (the #105588 follow-up report).
One seam after the lifecycle decision now clears agent._end_session_on_close
whenever the row was spared, replacing the gateway-owned-only assignment from
#111145 so the lease-held and Desktop-reclaim cases get the same treatment.
The five mocked _finalize_session tests from #105620 are replaced by two
real-DB _teardown_session tests beside the #111145 harness (red on both
origin/main and the bare cherry-picks, green on the stack).
ws_orphan_reap, idle_timeout, lru_evict, ws_disconnect, and tui_shutdown
are runtime/connection GC — not user intent. Ending the durable session
row during these automatic cleanup reasons confuses GC with user action,
causing secondary bot chats to vanish from the sidebar even though the
transcript is intact in state.db.
The canonical Bot Chat already has resurrection paths for accidental
ws_orphan_reap ends, but non-canonical secondary chats do not, so they
are hit harder.
Skip db.end_session() when _desktop_automatic_cleanup is True (automatic
cleanup reason + Desktop source). Runtime is still reclaimed, the
session.reclaimed event still fires, and explicit user close/archive/reset
still ends the durable row normally.
Fixes#105588
Two parametrized invariant tests (native client sync/async; None or raising
profile falls back to the standard client) replace four near-duplicates, and
the hook's kwargs are pinned exactly to the mapping openai.OpenAI would have
received. The fixture isolates both provider registries through monkeypatch
instead of a hand-rolled snapshot/restore, drops the HERMES_HOME override that
tests/conftest.py already provides, and replaces the tuple-truthiness lambda
with a plain function. Call-site comment trimmed to the ordering WHY.
The auxiliary api_key branch built openai.OpenAI directly, so an
out-of-tree provider registered with auth_type="api_key" lost its
native transport for auxiliary tasks even though the main-agent path
(_provider_supplied_client) and the external_process branch honor the
same hook. Consult the profile's create_client() before the built-in
gemini/OpenAI ladder; None (the default) falls through untouched, and
a raising profile is logged and skipped (#112384).
A `chat -q` worker's stdout/stderr go to its per-task log, so when it exits
without a terminal board call the reason is usually right there: the model's
explanation of why it could not comply (#88603) or the rendered provider error
(#46593). The reap discarded it and stamped a canned "protocol violation" /
"pid N exited with code C" on every retry. `_worker_final_output` reads the log
tail (trimming the CLI exit summary, Rich panel chrome and the session_id
trailer) and folds it into `last_failure_error` and the reap event payload as
`worker_output`, for clean exits AND crashes; `board` is threaded from the
dispatching tick so non-default boards find their own log directory.
Ported from #88815 (chelsealong) onto the decomposed dispatcher; widened to the
crash branch. Earliest attempt at the symptom: #46985 (joyjit).
Workspace moves stamp agent.session_cwd so a lazily started Codex thread
starts in the moved-to directory. Agents that never had the attribute
(test doubles, slotted objects) must keep working, so stamp only when the
attribute exists. Test stubs of _set_session_context mirror the new cwd
kwarg, and the Codex gateway test asserts the contract (no pinned session
cwd) instead of the attribute's absence.
Salvage of #93452 (@outpoints). Keep one invariant per fix:
resolver-level "automatic title never remaps a strategy session",
integration "provider routes by logical workspace, not process cwd",
agent-level "title provenance + cwd reach the provider", deferred
Desktop/TUI build threads the session cwd, seeded branch titles are
derived, and the workspace-move E2E. Drop the plumbing/legacy-shape
tests that re-assert the same contract.
acp_adapter: AIAgent(cwd=...) now stamps session_cwd itself, so the
direct assignment after construction was a duplicate.
Thread logical session cwd through deferred Desktop/TUI builds, normalize absent cwd during construction, and share title provenance constants between SessionDB and Honcho.
(cherry picked from commit 2693f4f27c776ac819d92c9b52e8a03ad2a985d8)
The Honcho provider resolved per-repo/per-directory session names from
os.getcwd(), which is the backend process launch directory on
Desktop/gateway hosts (typically $HOME), not the user's workspace. With
a manual sessions map entry for the home directory, every Desktop
conversation landed in that fallback bucket instead of the project's
per-repo session.
Use agent.runtime_cwd.resolve_agent_cwd() — the same single source of
truth already used for system-prompt and context-file discovery — so
Honcho session routing agrees with everything else about where the
agent logically lives. It honors the pinned session cwd, then
TERMINAL_CWD, then the launch directory; CLI sessions launched inside a
project resolve identically either way.
Adds a regression covering the Desktop-style case: backend launched in
$HOME, workspace elsewhere, home-directory manual map present.
Refs #24740
(cherry picked from commit cfb32757de2509dd404acf98311e357d73fa741e)
Auto-generated display titles (LLM or derived) were passed to Honcho's
resolve_session_name() as authoritative, so a titled per-repo,
per-directory, or global session silently remapped onto a second Honcho
session named after the generated title. Only explicit /title commands
(user provenance) should act as an intentional session-name override.
Thread session_title_source from the session DB through
agent_init into the Honcho provider, and skip title-based remapping
when the source is 'derived' or 'llm'. Missing provenance keeps the
legacy explicit-title behavior for callers that predate source
threading. Gateway per-chat keys and per-session identity safeguards
are unchanged.
Adds regressions for titled per-repo, per-directory, and global
sessions at both the resolver and provider level.
Fixes#24740
(cherry picked from commit e7ba26ee15821baa382a397ce9ce9cd57a260188)
`server_error` and `timeout` join the transient-provider set that makes a Kanban
worker exit 75 (EX_TEMPFAIL). A provider outage or a hung connection says
nothing about the task, so the dispatcher requeues without a failure tick
rather than counting toward the circuit breaker (#91206 proposed the same set).
scripts/ci/check_os_marker_fakes.py flags test files that make the
interpreter believe it is on macOS (is_macos -> True, sys.platform ->
"darwin", platform.system -> "Darwin") while carrying no `macos_only`
marker: the macOS lane imports only marked files, so such a file is green
on Linux over a faked branch and never runs on the host it exists for.
A `# os-marker: ok — <why>` comment opts a host-independent line out.
_BASELINE holds the files that already faked macOS when the check landed;
a stale entry fails the check so the list can only burn down. Wired into
lint.yml next to the compat-pointer check; two invariant tests cover a
flagged fake vs marked/opted-out files and host-honest platform reads.
The autouse fixture neutralised gateway discovery and the systemd branch
but not the launchd one. On a macOS host `_restart_macos_launchd_gateways`
derives its labels from the profile layout, so a default profile alone
hands it `ai.hermes.gateway`, the label never "comes back", and nine
unrelated update tests exit 1 with "Update incomplete". No OS is faked:
the seam is stubbed the same way test_update_fleet_restart_pending does.
The code registry cannot tell a plugin-only name from one the gateway
reads straight off os.environ (TELEGRAM_GROUP_ALLOWED_USERS), so the
note was false for real settings. Every UPPER_SNAKE name simply lands
in .env; the docs say so.
Production code must not depend on website/docs being present; the
"Hermes does not read this name" note now keys only on the code
registry (OPTIONAL_ENV_VARS / _EXTRA_ENV_KEYS / setup-hidden suffixes).
`hermes config set TELEGRAM_GROUP_ALLOWED_USERS ...` (and ~290 other documented
variables Hermes reads straight from os.getenv without registering them in
OPTIONAL_ENV_VARS) still landed as a config.yaml top-level scalar with a notice,
while the setup flows write .env and one-shot CLI readers never bridge YAML
scalars — two writers, two readers. #112250 routed the registered names; this
closes the class with a shape rule: any bare ^[A-Z][A-Z0-9_]*$ key is an
environment setting.
- set: writes .env, drops a stale config.yaml copy, never writes UPPER_SNAKE
into config.yaml (--force included); the env writer's denylist
(HERMES_YOLO_MODE, PATH, ...) now refuses cleanly instead of the YAML detour
bridging the value into os.environ; a name neither registered nor in the
environment-variables reference gets a one-line note but is still saved.
- get: .env first; a leftover top-level config.yaml copy is reported as stale.
- unset: removes the .env entry and the stale copy.
- Registered names, credentials (credential lifecycle + masking), dotted paths
and lowercase bare keys are unchanged.
Fixes#111848 (first half landed in #112250).
A clone can land unreadable (Windows ACL inheritance -> WinError 5, a
mode-000 file). Discovery now skips such a dir instead of aborting
(#112293), but the install that produced it still exited 0, so the user
got a plugin that silently never loads.
After the clone and before anything moves into place, walk the staged
tree and open every file / list every dir. On failure repair u+rX where
the OS honours mode bits; if still unreadable raise
PluginOperationError naming the file and the fix (icacls / chmod). The
staging dir is cleaned up, nothing is installed, exit is non-zero.
Fixes#111804 (its discovery half landed in #112293).
The non-quiet one-shot path exited 0 unless a Kanban worker was running, so
scripts could not tell a failed `hermes chat -q` from a good one and an
incomplete turn (partial, iteration budget) still read as success (#111770).
Both one-shot paths now share one contract: 0 completed, 1 failed / partial /
incomplete / never ran, 130 interrupted. The Kanban EX_TEMPFAIL sentinel also
fires for `upstream_rate_limit` (aggregator's upstream 429) and `overloaded`
(503/529): neither says anything about the task, so the dispatcher should
requeue without a failure tick rather than count it toward the breaker.
When the CLI approval callback raises, when no callback is registered on the
thread while prompt_toolkit owns the terminal, or when the input() read is
interrupted, prompt_dangerous_approval returned "deny" and the command gate
rendered "BLOCKED: User denied this command" — attributing a refusal to a
user who was never asked (#22992). #112308 fixed the gateway half of the
class (withdrawn prompts -> outcome "cancelled" with a cause); this closes
the CLI residual on the same shape.
- tools/approval_prompt.py: those three paths return an Unanswered("cancelled")
sentinel carrying the cause; MCP elicitation consent maps it to "cancel".
- tools/approval.py: the CLI gate renders "BLOCKED: <noun> was not approved: the
approval prompt could not be delivered or was not answered (<cause>)" with
outcome "cancelled" — still fail-closed, "Silence is not consent".
- tools/file_tools_write_guards.py: the protected-instruction write gate
reports the undelivered prompt instead of "was denied by the user".
- Shared metrics: "cancelled" is a counted approval outcome (contract + v2
schema) instead of falling into "unknown".
- Docs: hook `choice="cancelled"` now covers the CLI causes.
Fixes#22992
The per-session notification poller ran `_poll_bot_live_delivery_once` every
0.5 s. Once a "Bot Chat" session exists, each pass opens state.db and takes
the exclusive active-session registry lock; on Windows (`msvcrt` LK_LOCK
gives up after 10 s of contention) that raised
`RuntimeError: active session file lock unavailable` and the loop logged
`Bot live-owner delivery poll failed` on every attempt — 8,838 warnings in
three days, 91% of one install's WARNING output (#111719).
Two changes:
- `tools/bot_live_delivery.has_mailbox`: the mailbox directory is created
only when a delivery is first admitted, so a profile without it has nothing
to claim — the poll now returns before the state.db open / registry lock.
This keeps cron→Bot Chat and Bot Mode DM delivery intact on installs with
no messaging platform configured (both deliver through this mailbox), which
is why the poll is gated on the mailbox rather than on connected platforms
(PR #111733's guard would have broken those).
- `_poll_bot_live_delivery_guarded`: a failing poll backs off 5 s before the
next attempt and is logged at WARNING once per 60 s window (with the count
of suppressed repeats), DEBUG otherwise.
Live probe (real poller loop, temp HERMES_HOME with a Bot Chat row and the
registry lock made unavailable, 3 s):
before: owner_lookups=6 WARNING=6 (with or without a mailbox)
after: no mailbox -> owner_lookups=0 WARNING=0; mailbox -> owner_lookups=1 WARNING=1
Fixes#111719
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Salvage of #93649 (@TurgutKural). _apply_pending_fleet_restart_catchup took two
booleans (respect_no_gateway_restart + no_gateway_restart) that were only ever
true together; one `defer` keyword says the same thing. The 13 tests are cut to
the two invariants (pulled path skips restart + verify and keeps the marker;
already-current path defers the catch-up). The user guide gains a section on
running `hermes update` from inside the gateway.
The Kanban dispatcher spawns workers as `hermes ... chat -q <prompt>`
(`kanban_db.py::_default_spawn`). That path ran the turn and fell through
to an implicit 0 whatever happened — success, failure, or a provider
quota wall.
`detect_crashed_workers` reads rc=0 with the task still `running` as a
protocol violation, and protocol violations trip the breaker at
`failure_limit=1`, so a single HTTP 429 blocked the card permanently and
every card queued behind it stayed in `todo` forever waiting on a parent
that could never reach `done`.
`KANBAN_RATE_LIMIT_EXIT_CODE` (EX_TEMPFAIL) exists precisely to prevent
this: `_classify_worker_exit` maps it to a `rate_limited` kind and the
task is released back to `ready` without counting a failure. The consumer
end was complete and tested. The producer end was wired into the `-Q`
path only — the one the dispatcher does not use.
This extracts that mapping into `_single_query_exit_code()` and applies it
on both one-shot paths. `chat()` returns the rendered response string, so
the non-quiet path could not see the outcome; `_chat_settle_turn` now
records the raw turn result for it to read.
Scope is deliberately narrow. The non-quiet path only exits non-zero when
`HERMES_KANBAN_TASK` is set, so interactive runs and ordinary `hermes chat
-q` invocations still exit 0 exactly as before. For a dispatcher-spawned
worker the full contract now applies: 0 on success, 1 on failure, and the
sentinel on a rate-limit/billing wall.
Tests cover the path that was missed rather than the one that already
worked: 16 of the 17 new assertions fail on the parent commit, and the
key regression fails as `assert None == 75` — the exact rc=0 fall-through
— rather than on a missing symbol. The seventeenth asserts that a human's
one-shot run keeps exiting 0, and passes both before and after.
The plugins from the 2026-09-15 sweep that were held back for reasons that are
now resolved: Gondola failed only the validator bug fixed in 5f9042be;
hermes-intelligent-memory lives in plugin/intelligent_memory (subdir entry);
inkbox merged our provides_hooks fix (pinned at the merged commit);
home-dashboard and obsidian-memory are manifest-only dashboard plugins whose
probe skip was misread as a warning to hold on. All five validate ok in a bare
pip install -e . venv at the pinned sha; scanner verdicts safe/caution.
The salvaged commit shipped two tests: a disconnect-mid-stream test that reads
the reply back through GET /v1/runs/{run_id}, and a structural guard that
opened api_server.py and searched its text for `output=`. Tests that read
source are change-detectors, not invariants; the behavioural test already
fails the moment `output` disappears from the terminal write, so the guard is
gone. The inline comment shrinks to the WHY (parity with /v1/runs, the
recovery path) and drops the narrative.
`POST /api/sessions/{id}/chat/stream` mints a run_id and writes run status,
but its terminal write omitted `output=` — the reply text went only onto the
SSE queue. `POST /v1/runs` has always recorded it.
The asymmetry costs a caller its answer. A client whose socket dies mid-turn —
a sleeping laptop, a dropped WiFi link, a peer DM over a flaky LAN — sees the
run reach "completed" through GET /v1/runs/{run_id} and has no way to learn
what the agent said. Worse, the disconnect path interrupts the agent, which
unwinds and *returns* a partial result, so that partial answer is recorded as
a clean "completed" and then discarded: indistinguishable from a run that
produced nothing, and equally unrecoverable.
Record `output` on this route too. Terminal statuses are already retained for
_RUN_STATUS_TTL (3600s), so the existing GET /v1/runs/{run_id} becomes a
recovery path for any client that loses its stream, without a new endpoint,
without touching the run registries, and with no change to the streaming
contract.
Deliberately nothing else: the detached turn stays out of _active_run_tasks
(it is already counted via _inflight_agent_runs, and a task entry would
double-count it in the shutdown drain), and the run stays out of
_run_streams_created, so the orphan sweeper's 300s reap still cannot see it.
Tests: a disconnect mid-stream now leaves the reply readable both in the run
record and through _handle_get_run; a structural guard asserts BOTH routes
still pass output=, anchored on the status write rather than the SSE payload's
"completed": True key, so the asymmetry cannot quietly return.
The MoA aggregator and test stand-ins never merge extra_body; the bypass
must hand them the kwargs untouched or the conversation would be sent
empty. Fold that control into the existing rail test (still two tests).
Internal moves get no compat aliases (root AGENTS.md); codex_runtime and
auxiliary_client import bypass_sdk_request_transform from agent.sdk_transform_bypass.
The cherry-picked helper deleted 'tools' from the typed kwargs, so the SDK's
post-transform extra_body merge appended it after the caller's extra_body keys —
equal dict, different bytes (byte-keyed prompt caches would miss). keep_slots=True
leaves [] placeholders that the merge overwrites in place. Drop the invented
HERMES_CHAT_SDK_TRANSFORM env var; the pre-existing HERMES_CODEX_SDK_TRANSFORM
hatch from #93650 now disables both API families. Tests trimmed to the two
invariants (byte-identity incl. caller extra_body precedence; escape hatch).
`chat.completions.create` re-walks the whole request body against the
`CompletionCreateParams` union graph client-side, with the GIL held, before
any byte leaves the process. #93650 documented that class of walk wedging
for 12+ hours on a ~1.4 MB conversation: no in-process watchdog can fire
while the GIL is held, and no socket kill helps a pre-network hang.
through `extra_body`, which the SDK merges into the JSON body after the
transform — but scoped it to `responses.create`. The default chat path,
which every OpenRouter / Nous / xAI / DeepSeek / Kimi / llama.cpp /
Ollama / LM Studio / LiteLLM request takes, still pays the full walk.
Measured against a real `openai.OpenAI` over an `httpx.MockTransport`
(canned SSE, no network), with the request body captured from the
transport on both sides:
101 msgs / 76 KB 13.6 ms -> 1.2 ms
401 msgs / 190 KB 48.6 ms -> 2.0 ms
1601 msgs / 650 KB 188.7 ms -> 5.8 ms
and the bytes the server receives are IDENTICAL — literally equal, not
merely equivalent (194,894 == 194,894 at 401 messages). The cost is paid
per API call, so a tool-using turn multiplies it by its iteration count.
The three helpers move from agent/codex_runtime.py into a shared
agent/sdk_transform_bypass.py, re-exported under their original names so
agent/auxiliary_client.py and tests/run_agent/test_codex_sdk_transform_bypass.py
keep working untouched. The field tuple is now a parameter:
("input", "tools") for Responses, ("messages", "tools") for chat.
Two chat-specific details. `messages` is a @required_args parameter, so it
stays in the typed kwargs as an empty list and the extra_body copy
replaces it in the body — hence the new `required_empty` argument, which
Responses does not use. And the bypass is gated on the target actually
being the SDK's Completions: Hermes also drives chat-completions-shaped
facades that are NOT the SDK — the in-process MoA aggregator most
importantly — and those never merge extra_body, so handing them one would
silently send an empty message list. That guard is also why this needs no
edits to the 32 test files that assert on kwargs["messages"]: they mock
with stand-ins, not the SDK.
Every rail the merged PR was reviewed on is kept: the plain-JSON-only
guard so pydantic models and generators stay on the typed path, caller
`extra_body` precedence via setdefault (load-bearing here — the chat path
already populates extra_body from custom providers, reasoning config and
Nous Portal), and an env escape hatch, HERMES_CHAT_SDK_TRANSFORM=1,
mirroring HERMES_CODEX_SDK_TRANSFORM.
The summary/compression call sites at chat_completion_helpers.py:3449 and
:3514 carry the largest payloads in the process and are deliberately left
for a follow-up: they route through a lambda whose client is not in scope
at the call site, so they need a slightly different shape and a wider
test surface than this change.
fetchJson/fetchPublicJson only reached the redirect diagnostic when the 3xx
carried an HTML body or text/html content-type; an empty-body 302/307 (the
common reverse-proxy / forward-auth shape) still resolved null through the
earlier empty-body check. http.request never follows redirects, so classify
on status first and reject every 3xx with the redirect error regardless of
body.
Pass res.headers.location through so the message says where the request was
sent, and stop blaming credentials when the Location differs from the
requested URL only by scheme or trailing slash -- that is a saved-URL
mismatch, not an authentication proxy. Drop the 404 mention from the
docstring/test: both callers reject >= 400 before this branch, so only the
2xx leg carries the endpoint-missing capability wording.
Part of #112072
ouyangbo
brooklyn!