Port from Kilo-Org/kilocode#13224: fixed waits belong in foreground terminal calls, while background mode is reserved for independently running processes.
close() escalated to kill_process_tree() but never imported it; the NameError
was swallowed by contextlib.suppress, so on the TimeoutExpired path neither the
kill nor the post-kill wait ran and a root codex ignoring SIGTERM leaked (a
regression vs the previous self._proc.kill()). Import it from agent.deadline
and add a test forcing the timeout path that asserts the tree kill and the
follow-up wait both run. Also drop the upstream product reference from the
test docstring (credit stays in the PR body) and pass encoding= to the PID
file reads flagged by the Windows footgun scanner.
Port from openclaw/openclaw#126285: snapshot Codex app-server descendants before root retirement and sweep the proven process identities after close so independently grouped stdio MCP children cannot survive client shutdown.
A plain ``profile=None`` default could not tell "caller named the actor"
from "read the card" — an unassigned card is a legitimate None actor, and
the row re-read would silently kick back in for it. Use a module sentinel
so only callers that did not pass an actor fall back to the card row.
Trims the salvaged test file to two invariants in the existing review
lifecycle suite: the never-claimed handoff names the implementer (red on
main) and an unassigned card's synthesized run keeps ``profile=NULL``.
request_review captures the implementer before rewriting tasks.assignee to
the reviewer, but _synthesize_ended_run re-reads assignee off the mutated
row, so the zero-duration review_requested run names the reviewer instead
of the handoff's actor. Pass the captured implementer through a new
optional profile keyword on _end_or_synthesize_run/_synthesize_ended_run.
Fixes#111064
The export_all assertion already lives in tests/hermes_cli/test_session_export_batch.py. Replace
it with the lineage test (timings over merged messages, red on the previous head) which also
proves the derived block is excluded from the import size budget.
export_all rows now carry a computed `timings` key; the batching test built
its expectation from search_sessions + get_messages, so equality failed on
the extra key (CI red). Strip it for the row comparison and assert it is
present on every row.
`_export_timings` parsed message timestamps with a bare `float()`, so a
pre-existing out-of-range cell (e.g. 8.4e252) reached
`datetime.fromtimestamp` and raised OverflowError, killing the whole
`export_session`/`export_all` call — a regression caught by
tests/hermes_state/test_corrupt_row_robustness.py. Route the cell through
`coerce_epoch`, the reader every other timestamp surface already uses, so a
bad row degrades to a `missing` count (with the usual warning naming the
session) and the export still lands.
Seven near-identical predicate tests collapse into one invariant test
(flag OR sentinel name; neither on a normal channel; no flags attribute
is not obfuscated) plus the directory-filter test, per the repo's
<=2-invariant-tests-per-fix rule. The explicit-id backfill path gets its
own regression test in the existing backfill harness (fails on the
unfixed code, verified by sabotage).
Discord's Channel Obfuscation change (announced Aug 12 2026, HTTP
enforcement Nov 16 2026) dispatches channels the bot lacks VIEW_CHANNEL
on with name "___hidden___", flag 1 << 17 (CHANNEL_OBFUSCATED), and
nulled fields. Without filtering, the channel directory lists phantom
"___hidden___" entries the agent can never post to, and wildcard
missed-message backfill wastes history reads on channels that always
403.
Adds is_discord_channel_obfuscated() to gateway/platforms/helpers.py
(checks the flag bit plus the sentinel name for discord.py builds that
don't expose the new flag) and applies it at both enumeration sites.
Meta delivers non-conversational payloads on the messages webhook
field: type=system (user_changed_number, and since Aug 11 2026
user_changed_user_id BSUID-rotation events), type=reaction (emoji
taps), and type=unsupported/unknown. The Cloud adapter's event builder
fell through its type dispatch for all of these, defaulting to
MessageType.TEXT with empty body — producing a MessageEvent with no
text that kicks off a blank agent turn (and records typing/read
receipts against a non-message).
Drop these envelope types before any processing, mirroring the
inbound-contentless filtering the Baileys and Matrix adapters already
do.
Greptile's two findings on the original PR were both right.
1. The scaler read test_durations.json from the checkout, but CI ran on
a fresh runner where that file never exists (it is gitignored and the
slicing-era artifact/merge job that produced it is gone). The feature
was inert exactly where the false FLAKY kills happen. tests.yml now
restores the most recent main-saved cache before the run (PRs read
only) and saves it after a green push to main, mirroring the
ci-timings-baseline restore/save pattern already in ci.yaml.
2. _save_durations persisted every file's total subprocess wall,
including the ~cap of a timed-out attempt and the retry-summed wall
of a FLAKY file. With the scaler that compounds: a hang cached at
~300s earns 900s next run, then ~900s cached earns 2700s, until the
job timeout is the only bound. _clean_pass_durations drops failed and
FLAKY files from the write so a file's cached duration is always a
first-attempt-clean measurement; those files keep their previous
known-good entry.
Tests trimmed to the salvage bar (<=2 invariants for the scaler plus one
for the cache filter) and moved next to the other runner tests under
tests/scripts/.
The flat 300s --file-timeout SIGKILL'd known-slow large-collection
files when CI load dilated their runtime past the cap; the automatic
one-shot retry then passed, manufacturing a FLAKY report for a healthy
file. Seen 2026-08-18 on main run 32155223248's sibling PR runs:
tests/test_hermes_state.py (239 tests) killed at 300s on attempt 1,
passed in 205s on retry.
_effective_file_timeout() now gives each file
max(flat_cap, 3 x last cached duration) from test_durations.json.
The bound is only ever raised — genuinely hung files are still killed,
uncached files keep the flat cap, and --file-timeout/HERMES_TEST_FILE_TIMEOUT
semantics are unchanged.
Includes a sabotage-verified unit test (fails without the scaler).
tasks.worker_pid outlives a reboot; afterwards the number can belong to any
process. _pid_alive answered from bare existence, so reclaim_stale_claims kept
extending the claim of a "live" stranger (stuck running task), and
enforce_max_runtime / _terminate_reclaimed_worker SIGTERM'd then SIGKILL'd it.
_set_worker_pid now records gateway.status.get_process_start_time(pid) as
tasks.worker_started_at (additive column, NULL on legacy rows). _worker_alive
(pid, started_at) is the liveness check every reader uses (reclaim, defer,
reconcile, crash sweep, max-runtime, archive, reopen invalidation); a live pid
whose fingerprint disagrees is a recycled PID: treated as dead, never
signalled (termination reports pid_recycled). Legacy rows without a
fingerprint keep the existence answer until their next spawn.
Row hermes_cli/kanban_db_dispatch.py:458 (lane4_high) confirmed by tracing:
the SIGKILL at :461 was gated only on _pid_alive.
The profile scope (HERMES_HOME override, secret scope, terminal policy) is a
contextvar bundle bound per turn. A bare threading.Thread / Timer / gRPC
callback starts with an empty context and resolves the LAUNCH profile:
- agent/title_generator.py: the auto-title thread read
auxiliary.title_generation (model, language, provider key) from the default
profile's config and billed the default's key for a secondary's session.
Spawn via agent.memory_provider.spawn_context_thread (copy_context).
- tui_gateway/session_lifecycle.py: every teardown caller is a bare Timer
(ws-orphan reap), the idle-reaper thread, atexit _shutdown_sessions, the
session.close pool RPC, superseded_by_resume or compute_host flush - none
carries a scope, yet on_session_end / commit_memory_session / agent.close ->
shutdown_memory_provider read the provider's config + credentials at call
time. Under multiplex they failed closed (tail never committed, #110622
class); on the Desktop backend a secondary's transcript went to the launch
profile's memory tenant. _finalize_session and _teardown_session now bind
_session_profile_runtime_scope(session) around those blocks, which covers
every spawn site through the single chokepoint.
- plugins/platforms/google_chat/adapter.py: Pub/Sub callbacks run on the gRPC
SubscriberClient's threads and run_coroutine_threadsafe copies THAT empty
context onto the loop task, so _dispatch_message and everything under it
(attachment cache, per-user OAuth token store via _acquire_user_chat_api ->
_load_per_user_chat_api, TTS keys, delivery ledger, bot-id cache) resolved
the launch profile. connect() captures its scope; _on_pubsub_message and
_submit_on_loop run under a per-callback copy of it.
spawn_context_thread gains a kwargs passthrough for the title thread's
callbacks.
Under gateway.multiplex_profiles (and the Desktop/dashboard backend serving named
profiles) os.environ holds the LAUNCH profile's .env. Five spawn sites built a
child's env from it while acting for another profile, so the child saw the
launch profile's HERMES_HOME (bot_relay, key_cmd), its credentials, HERMES_MODEL
and TERMINAL_* policy, and none of the served profile's own .env:
- tui_gateway/server.py _SlashWorker: pinned HERMES_HOME but kept the launch
base with tier-2 credentials + settings.
- tools/bot_relay.py delivery_env (relay RPC + --run-delivery): dict(os.environ).
- tools/browser_tool.py _build_browser_env: re-added BROWSERBASE/FIRECRAWL/
BROWSER_USE keys from os.environ after the scrub.
- plugins/platforms/a2a/adapter.py _forward_to_profile: {**os.environ}.
- agent/command_token_source.py _mint: key_cmd helper inherited os.environ.
tools.environments.local.served_profile_child_env is the one builder: pin the
target home, drop the launch profile's .env residue and bridged TERMINAL_*
(strip_launch_profile_env), and for children that legitimately run with the
profile's credentials (agent worker, token helper) overlay the target profile's
own secrets - what a standalone `hermes -p X` loads itself, never a sibling's.
The browser keeps the provider scrub and re-adds only its passthrough keys via
get_secret. Outside multiplex the env is unchanged.
Live proof from inside the child (launch A, served B, multiplex on): all five
children print HERMES_HOME == B, see B_MARKER=b from B's .env and do not see
A_MARKER; the browser child gets B's FIRECRAWL_API_KEY. On base every one leaked
A_MARKER and lacked B_MARKER; bot_relay and key_cmd also had A's HERMES_HOME.
tests/tui_gateway/test_multi_profile_hosting_fail_closed.py — through the
registered RPC handlers: config.get for a secondary resolves only its own
secrets and flips the process fail-closed; llm.oneshot / model.options bodies
see the profile's home + secrets; a launch-profile agent build binds its own
scope once multiplexing is active; CONTROL: a single-profile serve keeps the
os.environ fall-through.
tests/hermes_cli/test_web_multi_profile_scope.py — through the real FastAPI
app: GET /api/config?profile=b expands only B's refs and never mutates
os.environ; console `send` for B lands B's .env in the request scope, not the
process env.
All red on origin/main by source swap (control green), green on head.
test_profile_terminal_scope_entrypoints follows the launch_profile_policy
rename and releases the launch turn's new secret scope.
tests/conftest.py resets the process-global hosting latch
(_MULTIPLEX_ACTIVE, the frozen launch env, _served_profile_homes) per test:
one request routed to a named profile otherwise left every later test in the
file fail-closed.
When a child's final answer still missed its output_schema after the one
bounded retry, the result entry flipped to status=failed with the error
"Final answer does not satisfy the declared output_schema" — the completion
line printed ✗ and orchestrators read a finished audit as a failure. Five
audits of 413-4103 s were lost this way in the Sep 10-14 retrospective and
the parent had to mine the live transcripts; in four of them the "violation"
was a ```json fence around a valid array, which the candidate extractor
sliced to its first..last object.
Now: status stays completed, `summary` is the child's raw final text,
`schema_valid: false` + `schema_errors` carry the verdict and a `schema_note`
says the text is unvalidated; the sync completion line shows ⚠ with the
reason. The extractor tries the earliest-opening bracket span and keeps the
first that parses (fenced arrays validate). The OUTPUT CONTRACT the child
sees now says "ONLY the JSON value — no prose, no code fence" and what a miss
costs. One bounded retry is unchanged.
HERMES_DELEGATED_CHILD_CONTEXT=1 is deliberately carried into every shell/
execute_code subprocess a delegate_task child spawns (the fence must survive
exec so a grandchild `hermes kanban complete` cannot promote itself). But the
readers treated the bare flag as "fence every Kanban DB": kanban_db_connect
opened ANY board ?mode=ro and write_txn refused ANY mutation. A subagent
running a Kanban reproduction against a scratch HERMES_HOME therefore got a
silently read-only board with a misleading "descendants require an
initialized board" error; only one lane in the retrospective ever discovered
why (deleg_15dac332), every earlier kanban repro ran degraded.
The marker's value is now the fenced board ROOT (kanban_home() at spawn) and
readers deny only paths under that root or the dispatcher-pinned
HERMES_KANBAN_DB (kanban_path_is_fenced). In-process children and a legacy
"1" marker still fence everything; an inherited path marker is never
re-derived, so a grandchild that moved HERMES_HOME cannot unfence the real
board. Owner-gate tests (test_kanban_descendant_scope, cron env isolation,
kanban CLI exit status) are unchanged and green.
A delegated child's execute_code kernel was keyed correctly
(<owner>::child::<session>) but counted against the process-wide
max_session_kernels LRU cap (default 4) like any other kernel. In a fan-out
wider than the cap every child's first cell spawned a kernel and evicted the
oldest sibling's, so the sibling's next cell started a fresh interpreter and
NameError'd on state its own previous cell had set — while the tool schema
promised "variables, imports, and loaded data survive across execute_code
calls". Finished children's kernels also squatted the cap for
kernel_idle_timeout (1800 s) after the child was gone. 48 NameErrors across 28
subagent lanes in the Sep 10-14 retrospective.
A live child's kernel (local and remote) is now pinned: exempt from LRU
eviction while the child runs, disposed by the delegation cleanup path
(shutdown_kernels_for_delegated_child) as soon as the child finishes. Top-level
sessions keep the existing cap and idle reaping unchanged.
_handle_session_expired_and_retry only reached the at-most-once guard when a
reconnectable server record existed; without one (server torn down, MCP loop
not running) a write-capable call fell through to the generic "MCP call
failed" error, which invites the model to replay a write that may already
have landed. The session-expired classification now runs first and a
write-capable call always gets the outcome_uncertain error; the reconnect is
attempted only when a server can be signalled.
_track_inflight_rpc's teardown RuntimeError said "retry the request on the
rebuilt session" for every op; for a write-capable tools/call it now says the
request may already have been dispatched and must be verified first, so the
wording matches the at-most-once contract the recoverer enforces.
Docs: the readOnlyHint row explains that the same hint gates auto-retry after
a mid-call session expiry, and that unannotated tools on an idle-TTL
Streamable-HTTP server return outcome_uncertain on the first call after idle
instead of being transparently replayed.
A 'session expired' / transport-closed failure can arrive AFTER the server
already accepted and executed the request (proxy-synthesized 404s, pod
rotation, ClosedResourceError firing mid-response). Auto-retrying a
write-capable tool in that window risks a duplicate side effect that MCP
offers no way to undo.
The session-expired recovery path now consults the discovery-time
readOnlyHint capture (same data the trust gate uses): only tools whose
annotation is exactly True keep the reconnect+retry-once behavior. Write-
capable calls still get the transport healed (reconnect, breaker reset on
success) but return a structured outcome_unknown error telling the model
to verify with a read before re-invoking.
The OAuth 401 path keeps its retry for all tools: a 401 means the server
demanded authorization before dispatch, so the call never executed —
matching the upstream classifier's McpAuthRequiredError => safe rule.
Fails safe: missing/unknown annotations classify as write-capable.
The per-process address-space cap was defense-in-depth on top of the
SessionDB leak sweep (nobody sets the knob; the sweep removes the leak).
On the 96-worker CI runner it was also the only PR-specific difference
when tests/tools/test_image_source.py hung to the 600s SIGKILL while the
same file passes in ~30s on every sibling branch: RLIMIT_AS counts virtual
reservations, and image/threading libraries reserve far more address
space than they touch. Keep the sweep, remove the cap and its env knob.
Root cause of the 2026-08-16 OOM incidents (three runs of
`python -m pytest -o addopts= -q tests/hermes_cli/` ballooning to
16-25 GB RSS and getting killed): ~40 files under tests/hermes_cli/
construct SessionDB() directly and never close it. Each instance keeps
the writer connection (state.db + -wal fds), up to _READ_POOL_MAX pooled
readers with their SQLite page caches, and — once token accounting has
run — an atexit registration that pins the instance alive until
interpreter exit. In one process over 637 files those accumulate without
bound; the sanctioned per-file runner masks it, so CI never saw it.
Fix the class, not the sites:
* hermes_state: register every successfully constructed SessionDB in a
test-only WeakSet (populated only when HERMES_TEST_ISOLATION is set,
i.e. under this test suite; production never touches it).
* tests/conftest.py: autouse _close_leaked_session_dbs teardown closes
everything left in the registry after each test. close() is idempotent
and unregisters the pinning atexit hook, so instances become
collectable.
* tests/conftest.py: session-scoped _pytest_memory_cap applies a
defensive RLIMIT_AS of 12 GiB (Linux only) so any future in-process
leak fails fast with MemoryError instead of eating the box.
Overridable/disable-able via HERMES_PYTEST_MEM_CAP (documented in
scripts/run_tests_parallel.py).
* tests/hermes_state/test_session_db_leak_sweep.py: behavior contract
for registration, idempotent close, and the cross-test sweep.
Measured (capped single-process `pytest -o addopts= -q tests/hermes_cli/`):
peak RSS 4.16 GiB before -> 1.67 GiB after; per-test open .db fd count
previously climbed monotonically (0 -> 12 -> 17 -> 104 within the
SessionDB-heavy files), now stays bounded (<= 5, transient). Sanctioned
runner over the affected 35 files: 495 passed, 0 failed, no FLAKY.
Incident evidence: ~/.hermes/logs/oom-incidents/20260816-202114
(fd dumps show 100+ open state.db/state.db-wal handles across pytest
tmpdirs; 3rd recurrence that day).
3fad83df31 (Aug 11) moved Relay exporter config to a plugins.toml
selected by HERMES_NEMO_RELAY_PLUGINS_TOML. A .env still carrying the legacy
exporter vars and no TOML logs ONE warning and initialises no exporters, so
users who followed the earlier docs lost every trace silently (the
maintainer's stopped Aug 20, noticed Sep 14; five multiplexed profiles on
the same box carry the same eight vars today).
- `hermes_cli/relay_plugin_migrate.py`: build the document from the
`nemo_relay.observability` dataclasses (`ComponentSpec(...).to_dict()`,
so the `type = "file"` sink discriminator is emitted), validate it by
activating it through `nemo_relay.plugin.initialize` + `clear_async`,
write `<home>/relay-plugins.toml` (tomli_w when installed, minimal emitter
otherwise), set HERMES_NEMO_RELAY_PLUGINS_TOML in that .env, and comment
the legacy lines out (never delete). Defaults mirror the removed plugin so
files land where they used to.
- `hermes update` runs it for the default home AND every live named profile
(each writes its own TOML) as a best-effort post-update step, with a loud
notice; `hermes migrate relay [--all-profiles] [--no-validate]` runs it on
demand.
- The runtime WARNING and the `hermes doctor` finding now say "NO traces
are being exported" and name the exact command and file path.
- Docs: environment-variables.md + built-in-plugins.md carry the migration
note and a complete plugins.toml example including `type = "file"`.
`_flag_reconnect_needs_attention` stamps needs_attention=True +
retrying_since when a reconnect loop passes the attention threshold, but
only `_install_reconnected_adapter` (the watcher's own success path)
cleared them. Every OTHER writer of `connected` — the startup stamp in
`_start_connect_pending`, `BasePlatformAdapter._mark_connected`, Telegram's
in-place polling recovery (`send_path_degraded` -> healthy) — left the
flags in place. A gateway restarted after an escalation therefore reported
telegram as `connected, needs_attention: true, retrying_since: 2026-08-30`
for two weeks on a healthy bot.
Fix at the single seam: `write_runtime_status(platform_state="connected")`
now defaults needs_attention=False / retrying_since=None unless the caller
passed them explicitly, so all four writers agree without each growing a
copy of the clear. Discord's connect() (cherry-picked from #102557 by
@sudhirpatil) calls `_mark_connected()` instead of a bare `_running = True`
so its stale `fatal` stamp clears through the same seam (#102554).
Under gateway.multiplex_profiles, `_start_one_profile_adapters` skipped
Platform.RELAY / Platform.WHATSAPP for secondaries with a bare `continue`,
and the startup "not being served" WARNING only covered platforms the
PRIMARY skipped. Four secondaries on one live box had WHATSAPP_ENABLED=true
and nothing in the log, status file, or `hermes gateway status` said the
channel was dead.
- `_note_unserved_secondary_platform`: one INFO per (profile, platform)
naming the reason (shared process-level ingress owned by the default) and
the remedy (enable it on the default profile, or disable it here), plus a
`<profile>:<platform>` runtime-status stamp (state=disabled,
error_code=multiplex_shared_ingress).
- `_start_secondary_profiles` folds those platforms into the loud WARNING
when NO profile (default included) runs them.
- `hermes gateway status --profile X` prints
`whatsapp: not served under multiplex (shared ingress owned by default)`
from that stamp; /api/status excludes `disabled` entries from the
platforms degraded verdict (informational, not a fault).
- Docs: multi-profile-gateways.md gets the shared-ingress rule.
DiscordAdapter.connect() set self._running = True directly instead of
calling self._mark_connected(), unlike every other platform adapter
(Telegram, WeCom, Matrix, Feishu, Google Chat, IRC, LINE, Mattermost,
ntfy, photon, raft, simplex, a2a, buzz, dingtalk, whatsapp).
_mark_connected() clears _fatal_error_code/_fatal_error_message/
_fatal_error_retryable and rewrites the runtime status file as
"connected". Bypassing it meant a transient connect failure (e.g. a
one-off DNS blip: "Cannot connect to host discord.com:443 ssl:default
[Temporary failure in name resolution]") left the platform reported as
permanently fatal in gateway_state.json / the dashboard, even after the
adapter successfully reconnected and was actively serving messages for
hours.
Reproduced under gateway.multiplex_profiles: true with a secondary
profile's Discord bot (sarathi:discord) — the bot reconnected
repeatedly ("Connected as ..." logged many times over 18+ hours) while
the dashboard kept showing the original fatal error the entire time.
Fixes#102554.
Added a regression test asserting connect() clears a previously
recorded fatal error.
config_loader._dm_behavior_choice still normalized against {"pair","ignore"},
so `unauthorized_dm_behavior: decline` in config.yaml (top level or a
platform block) was coerced back to "pair" on the real startup path
(load_gateway_config), and `unauthorized_dm_decline_message` was never
bridged into gw_data. Both now go through gateway.config.UNAUTHORIZED_DM_BEHAVIORS
(single source) and the presence bridge. The round-trip test exercises
load_gateway_config with a real config.yaml (top-level decline, telegram
override, custom message) instead of GatewayConfig.from_dict.
Telegram's intake prefilter only forwarded unauthorized DMs when the
behavior was exactly "pair", so with an allowlist configured a decline was
never sent. Anything that needs an outbound reply (!= "ignore") passes.
`hermes gateway setup` gains a "Politely decline unknown senders" choice
that writes platforms.<platform>.unauthorized_dm_behavior: decline; docs
mention it. Upstream-source references dropped from docstrings.
Port from qwibitai/nanoclaw#3260: adds a third unauthorized_dm_behavior
option, 'decline'. Instead of replying with a pairing code (pair) or
staying silent (ignore), the gateway sends one short, polite decline to
the unknown sender, then stays silent toward that sender for 24 hours.
- gateway/config.py: accept 'decline' in the normalizer; new
unauthorized_dm_decline_message for custom decline text (round-trips
through to_dict/from_dict).
- gateway/pairing.py: persisted decline stamps (_declined.json) on
PairingStore with has_recent_decline/record_decline; stamps are
pruned on write and recorded BEFORE delivery so a send failure can't
become a decline storm (nanoclaw's stamp-first pattern).
- gateway/run.py: decline branch in the unauthorized-sender path;
groups still always silently ignore.
- docs: security.md + configuration.md updated.
Adapted from TypeScript (NanoClaw's pending_sender_approvals 'decline:'
stamp rows) to Hermes' existing PairingStore JSON persistence; the
owner-FYI half of nanoclaw's flow is intentionally not ported — Hermes
logs the unauthorized attempt, and pairing remains the owner-visible
grant path.
Rebase onto the decomposed gateway (salvage, #88028):
- The unauthorized-sender path moved from gateway/run.py to
gateway/run_inbound.py::_hm_admit_event; the decline branch is a sibling
helper _hm_send_unauthorized_decline next to _hm_offer_pairing_code.
- gateway/config.py now validates the enum via _normalize_choice; the
accepted set is the module constant UNAUTHORIZED_DM_BEHAVIORS (used by
both from_dict and get_unauthorized_dm_behavior so a per-platform
`extra.unauthorized_dm_behavior: decline` is honoured too). The default
decline text lives in config as DEFAULT_UNAUTHORIZED_DM_DECLINE_MESSAGE.
- Tests trimmed from 5 to 2 invariant tests (send-once-then-silent through
the real inbound path; config round-trip + real PairingStore stamp
lifecycle with a patched clock instead of rewriting the JSON file).
remove_job() deletes <cron>/output/<job_id>/ together with the record, but the
finishing run then called save_job_output(), re-creating the directory and
writing the final run into it. Every self-removing job leaked an orphan
directory the store no longer knew about, and the docs claim that "only the
job record is gone afterwards" was false. Skip the save when
self_removal_delivery_allowed() is true (the same check that already excuses
the missing record on the delivery and mark paths); delivery composes without
an output_file, as it already does for non-file paths.
The BaseException handler in _run_one_job_body still called mark_job_run on
the missing record after a self-removal crash. Guard it with the same check so
the crash path matches the completion path instead of probing a deleted record.
Docs: state that the record and its output directory are both gone.
Review finding: self-removed run re-creates the rmtree'd output dir (orphan leak); crash path marks a missing record.
Follow-up to the salvaged #111044 commits:
- self_removal_delivery_allowed() now also requires that no record currently
holds the job id. The marker alone said "this run removed its record"; it did
not say the id is still empty. A replacement record (another owner reclaiming
the id) must be treated as a stolen claim, not a self-removal.
- Drop the allow_self_removed kwarg on fire_claim_fence: the fence already has
the job_id and the ContextVar marker, so it can decide on its own; the caller
no longer threads a flag it computed from the same predicate.
- _FireOwnership.lost(): keep the explicit lost event and the no-owner short
circuit ahead of the self-removal check so an interrupted run is still
reported as lost even after it removed its record.
- _finish_completed_run: skip mark_job_run entirely for a self-removed job
(nothing to mark) instead of calling it and then excusing the False.
- Tests trimmed to two invariants, both A/B'd against origin/main: the
self-removing run delivers after a post-removal heartbeat tick (RED on main),
and a self-removal followed by a replacement record is still discarded
(GREEN on main, guards the new predicate).
- Docs: user-guide cron.md notes that a job may remove itself and still report.
A run that deletes its own job after the first heartbeat interval was still marked stale because the fire-claim loop treated a missing record as lost ownership before the self-removal marker could win.
Co-authored-by: Cursor <cursoragent@cursor.com>
The salvaged parametrized set collapsed to one neutral case (transient) plus
the needs_input positive control. hermes kanban block now mirrors the
notifier: 'needs a human decision' only when the block was typed
needs_input, 'orchestration attention needed' otherwise.
A repeated-block circuit breaker routes a task to triage and establishes that
orchestration attention is needed — it does not establish that a human
decision exists. The notifier unconditionally rendered every
block_loop_detected event as "needs a human decision", overclaiming owner
intent for dependency waits, capability gaps, and transient failures.
Branch on the typed block kind in the event payload: only needs_input (the
one kind carrying a concrete question for the owner) keeps the decision
wording; dependency/capability/transient/None get neutral orchestration
wording. TRIAGE visibility, the reason, and recurrence count are preserved.
Closes#111125
The PR added four test functions for one fix. The host-side short-write and
multibyte cases are the same invariant as the sandbox size probe (an archive
that is not byte-exact is never referenced to the model), so they become two
parametrized rows of test_size_probe_decides_lossless, which now also uses
multibyte content so every row pins the byte-vs-char comparison. Net new
tests for the fix: 2 functions.
Also names in _write_to_sandbox why the +1 tolerance is keyed on heredoc
mode only: the payload backend delivers stdin verbatim and is expected to be
byte-exact. The three bare write_text() calls the footgun scanner flags in
this file gain encoding= while it is being touched.
Five near-identical MagicMock scripts pinned the same contract (byte-exact
or discard, heredoc gets exactly one extra byte); one parametrized test
plus the unprobeable-backend case cover it. Drop the host-side happy-path
test that duplicated test_multibyte_content_verified_by_byte_count.
Docstring now names the real heredoc wrapper
(BaseEnvironment._embed_stdin_heredoc) and notes the extra exec RTT per
oversized result (review point).
Oversized tool results are archived to disk and replaced in-context with a
'Full output saved to: <path>' reference. Until now the write was trusted
blind: a partially-flushed host file (ENOSPC/quota races) or a lossy sandbox
write (API-body truncation on payload backends) still produced the archive
reference, so the model was told the full result was recoverable when bytes
had silently vanished.
Both persistence paths now round-trip-verify size before building the
reference and fail closed to the bounded inline truncation otherwise:
- _write_to_spillover: byte-count check via os.stat after write; mismatched
archives are deleted and the caller falls through to inline truncation.
- _write_to_sandbox: wc -c probe after the cat; heredoc-mode backends get a
+1 byte tolerance (wrap_modal_stdin_heredoc appends one newline by
construction), unprobeable backends stay best-effort success.
Regression tests fail without the fix (verified by stashing the source
change: 4 failed). E2E-verified against a temp HERMES_HOME with real file
I/O including multibyte content and a simulated short write.
The PR's race test slowed save_config and hoped for an interleave, and
called the six handlers as web_server attributes that no longer exist
(web_server.py is a facade over web_routers/). Its second test read
inspect.getsource() for the lock name — a change-detector on source text.
Replace both with two invariant tests that drive the real endpoints via
TestClient and FORCE the interleaving: writer 1 is held inside save_config
until writer 2 has run load_config (or the lock kept it out). Unlocked the
second save erases the first mutation; locked the writes serialize.
Verified: reverting hermes_cli/web_routers/ to main fails both tests
(KeyError 'providers' / 'aggregator' = the lost write), the fix passes.
Only PUT /api/config took the span lock; model.set, moa, custom-endpoint
create/activate/delete, memory-provider saves, and the profile-dir model
write ran load→mutate→save unlocked in worker threads. The desktop fires
these concurrently with its debounced autosave — whichever save landed
second silently dropped the other's mutation (#88913/#89184 lost-write
flavor). Race test with slowed save proves both writes now survive.
`model.key_env` is not custom-only: the Desktop settings UI stores REGISTRY
provider keys there (e.g. HERMES_CUSTOM_LMSTUDIO_API_KEY with provider
lmstudio, #106336) and auth._model_level_key_env honours it. The previous
predicate (`not custom or route_changed`) therefore wiped that pointer on a
same-provider same-base_url model re-pick and silently broke the user's
credential. The pointer now clears ONLY when provider or base_url changed;
the inline api_key/api rule is unchanged.
Custom-endpoint activation (model_setup_flows_custom) popped base_url /
api_key but never key_env, so a stale pointer from a previous endpoint
outranked the credential it had just written — pop it alongside.
Tests: the same-route re-pick case now covers a registry provider (red on
the old predicate), and the two clear_model_endpoint_credentials tests are
folded into one invariant.
Main no longer clears credentials in web_server's _apply_main_model_assignment;
every /model surface (CLI, gateway, TUI, dashboard) persists through
model_selection_config_updates(), which only dropped api_key/api on a route
change. Custom-endpoint activation writes model.key_env with NO inline key, so
a pointer-only model block survived every provider switch and routed the new
provider's requests to the old endpoint's env var (the PR's Bug 5) — live
repro on main: activate custom_myep -> /api/model/set openrouter left
key_env: CUSTOM_MYEP_API_KEY on disk.
Port the PR's web_server hunk to the one shape function: key_env/api_key_env
clear under the same route-changed rule as api_key (same-route re-pick keeps
them). The dashboard's _resolve_assignment_credentials re-adds the TARGET
provider's own pointer after this, so custom->custom still ends with the new
endpoint's key_env. One invariant test in the one-shape suite.
Custom-endpoint activation writes model.key_env, but
clear_model_endpoint_credentials never popped it and the switch-clears
trigger only fired on inline keys — a pointer-only model block survived
every provider switch, routing the new provider's requests to the old
endpoint's env var. key_env/api_key_env now clear under clear_api_key;
the trigger fires on pointer-only blocks too. All key_env writers set it
after the clear, so custom-to-custom switches keep the new pointer.
- codex_runtime._CODEX_PROGRESS_DELTA_TYPES gains response.refusal.delta so the
stream watchdog sees progress on a refusal-only stream instead of timing it
out as idle.
- auxiliary_client._parse_codex_final_response reads type=refusal content
parts; without it an aux refusal-only turn parsed to content=None and hit the
empty-response path the main loop was just taught to avoid.
- tests: parametrize test_streamed_refusal_accumulated (refusal-only /
alongside-content) so there is one test per surface; drop upstream product
references from docstrings (credit stays in the PR body); pass encoding= to
the read_text calls flagged by the Windows footgun scanner.
- docs: fallback-providers notes that a streamed refusal is a terminal
content_filter result, not an empty response to retry.
A model that declines mid-stream delivers the explanation on the
structured refusal channel (chat_completions delta.refusal; Responses
response.refusal.delta / refusal content parts) and leaves content
empty. The streaming accumulators dropped that channel entirely, so a
streamed refusal assembled into an empty message and fell into the
empty/invalid-response retry loops - burning paid retries reproducing a
deterministic refusal - while the non-streaming path had already fixed
this class in #46013.
- chat_completions streaming: accumulate delta.refusal (incl.
model_extra), expose message.refusal on the assembled mock response so
ChatCompletionsTransport.normalize_response applies the existing
sole-payload -> content_filter promotion; count refusal deltas in the
zero-chunk guard; carry refusal in the Relay final-response dict.
- Codex Responses stream consumer: collect response.refusal.delta as
answer text so a refusal-only stream no longer raises 'did not emit a
terminal response' with zero usable content.
- Responses normalizer: read type=refusal content parts in
_extract_responses_message_text (attr and dict shapes).
Sabotage-verified: each new test fails with its wiring line disabled.
E2E: refusal-only stream -> terminal content_filter with explanation;
refusal-alongside-content stays a normal usable turn; plain-text
streams unchanged.
`_provider_pip_dependencies` still read ~/.hermes/hindsight/config.json with
strict utf-8 inside a bare `except Exception`, so a Windows-editor BOM made the
`mode` lookup silently fail and `hermes update` reinstalled only
`hindsight-client`, leaving the embedded daemon broken — the exact #70636
symptom this helper exists to prevent. Route it through the shared
`read_json_or_empty` (utf-8-sig, {} on missing/corrupt) like the other memory
readers in this PR, and add the reader to the parametrized BOM invariant.
Teknium