claim_job_for_fire refused any claim younger than FIRE_CLAIM_TTL_SECONDS (300s)
regardless of whether its owner was still alive, so a `hermes cron run` killed
mid-flight (timeout, Ctrl-C, OOM) blocked the next manual run with "already being
fired" for up to five minutes. The executions table already reaps dead owners on
sight; the job-record fire_claim now does the same via gateway.status._pid_exists
when the claim's `by` names a pid on this host. Foreign hosts, explicit
HERMES_MACHINE_ID values, and probe failures keep the TTL (fail safe).
#106582 (1c6683e8e0) added display_identity and display_order to the messages
table in SCHEMA_SQL without appending the matching events to
SCHEMA_HISTORY["messages"], so test_replayed_history_ends_at_current_schema
has been red on main since 17:06Z. Append the two '+' events; no runtime change.
#106623 (ca16cafee4) blocks any test subprocess whose argv resolves to
`hermes gateway run|start|restart`, so the harness cannot spawn a runtime that
outlives the worker and restarts the developer's gateway. The matcher strips the
`-p <profile>` selector and reads through `sh -c`, so it also fired on
`docker exec -u hermes <ctr> sh -c 'hermes -p x gateway start'` and broke
tests/docker/test_profile_gateway.py on every Docker build since. A gateway
launched inside a container cannot reach the host unit or webhook port; the
guard now skips commands whose argv[0] is a container runtime (docker, podman,
nerdctl). Host-side spawns stay blocked.
_is_azure_foundry_responses matches the azure-foundry provider or the
project-scoped services.ai.azure.com host. #101243 proposes narrowing it
to that host alone. The multi-item rejection this branch handles happens
on resource-level *.openai.azure.com hosts too, so the pruning gate now
uses _is_azure_responses: the provider, or either Azure host.
Refs #105369
The first request of every turn after the first replays the encrypted
reasoning item of every earlier response. Azure Foundry accepts one such
item and rejects two or more with HTTP 400 "Conflicting authenticated
continuation identities". In-turn requests never hit this because
_is_post_tool_replay already suppresses replay on Azure when the request
ends on a tool result.
On Azure Foundry, build_kwargs now keeps codex_reasoning_items only on the
newest assistant row that has any. include still requests
reasoning.encrypted_content, compaction checkpoints stay on every row, and
the canonical messages are not mutated. Other Responses endpoints are
unchanged.
Rebuilding a failed session from state.db through the transport and
sending it live: unpatched, 5 reasoning items, 400; patched, 1 item, 200.
Fixes#105369
Azure Foundry (gpt-6-astra) answers a request that replays encrypted
reasoning from more than one prior response with HTTP 400 "Conflicting
authenticated continuation identities", code invalid_value. The classifier
mapped that to a non-retryable client error, so the turn aborted and every
later turn in the session failed the same way.
Map the message to invalid_encrypted_content. The existing recovery in
turn_recovery then strips the replay state and retries once.
Refs #105369
The Desktop settings UI persists a registry provider's API key as a
credential pointer -- model.key_env in config.yaml pointing at an env var
in $HERMES_HOME/.env (e.g. HERMES_CUSTOM_LMSTUDIO_API_KEY) -- while
keeping model.provider on the registry id. _resolve_api_key_provider_secret
resolved registry providers exclusively from
PROVIDER_REGISTRY[...].api_key_env_vars, so the UI-saved key was silently
ignored; for lmstudio the runtime then substituted the no-auth placeholder
(dummy-lm-api-key) and auth-enabled LM Studio servers rejected every
request with an opaque, retryable-flagged 401.
Fix: consult model.key_env (only when config.yaml's main model targets the
provider being resolved, so the pointer never leaks across providers)
before the registry env vars, running the value through the same
_usable_declared_secret prefix validation. Precedence for the canonical
env vars and the credential pool is unchanged; a valid canonical key still
wins when no pointer is set.
Complements #75373, which covers providers.<id>.key_env at the provider
level; the Desktop save path writes the pointer at the model level, which
that PR does not consult.
Fixes#106336
Carry the one hunk PR #106424 adds over #106405: the 4001 message names the
door that deliberately changes the profile default, so a stale client's user
sees where to go instead of a bare "requires a live session". The flag
pre-parse from #106424 is not carried: with #106405's guard, sessionless
`--once --global` already surfaces the canonical conflict error through
_apply_model_switch (verified in-process).
Co-authored-by: mkrb84 <192544766+mkrb84@users.noreply.github.com>
Drop the injectable `topEdgePx` knob (one caller, one constant) so the
threshold has a single home, and fold the two-branch tail of
shouldAutoShowEarlier into one predicate. Comments keep the WHY: wheel is
the only signal at a clamped scrollTop 0, and the gates name the states
where scrollTop is near 0 without meaning "read earlier".
Tests: collapse the 11-case matrix into one invariant test on the pure
gate, and add a rendered-list test that mounts the real Thread against a
transcript heavier than RENDER_BUDGET, escapes the bottom lock, wheels up
at the clamped top and asserts more turn groups mount — while an upward
wheel mid-transcript mounts none. Red on origin/main (28 groups stay 28),
green with the fix.
Long conversations and branched sessions hide older turns behind the
render budget, but only the Show earlier button loaded them. Auto-page
through the same showEarlier path when the reader is at the viewport top.
Co-authored-by: Cursor <cursoragent@cursor.com>
Salvage bar is at most two invariant tests per fix. The monotonic-watermark
case (small, oversized, small -> is_truncated(s, 1)) is the same contract the
oversized-frame test already pins, so it lives there as a final block instead
of a third test function.
transport.write serializes outside its lock so one large payload cannot block
other threads' frames; the replay ring is on the same hot path (write_json), so
its sizing json.dumps must follow the same rule instead of holding _replay_lock
for the whole encode. seq is not stamped at that point, which is a few bytes off
a MiB budget and irrelevant.
Also restore the original WHY for the count limit, explain the byte ceilings
(512 x 64 KiB tool results is ~32 MiB per session before the cap; 4 MiB per
session, 64 MiB per process) and make the module docstring state the bound in
bytes and the truncation watermark.
The advisory step ran `git merge-base origin/main HEAD` where HEAD is the
checked-out refs/pull/N/merge commit. GitHub recomputes that synthetic merge
commit whenever main moves, so by the time the job runs the checked-out SHA is
often no longer reachable from any ref. `git fetch --deepen` on an
unreachable commit can never pull in its parents, so all four fetches in the
loop (200 + 3 x 1000) ran to completion, burned 4m15s, and still ended in
"public-surface: no merge-base" (run 34285107265, job 102259026802). The
step-level timeout keeps that overrun from cancelling the blocking job; this
commit makes the step finish on the first fetch instead.
Fetch refs/pull/N/head explicitly (always reachable server-side) and diff
that ref against the base; the merge commit's own content is irrelevant to a
symbol diff.
Raise the step timeout from 2 to 3 minutes: one --deepen=200 fetch took ~56s
and a --deepen=1000 ~67s on the runner, and main gains ~170 commits a day, so a
branch a day old legitimately needs both. 45s of blocking steps + 3 min still
fits the 5-minute job budget.
The 'Public-surface diff vs base (advisory)' step already carries
continue-on-error: true, so its own failure never fails the Windows-footguns
job. But continue-on-error does not shield the job-level timeout-minutes: 5:
when the deepen/fetch loop runs long (a distant or missing merge-base), the
step eats the whole job budget and the job is cancelled at ~5m even though the
blocking footgun/compat checks passed and the public-surface check is advisory.
Give the advisory step its own timeout-minutes: 2. A step overrun is then
killed and, via continue-on-error, kept off the job outcome the same way a
step failure already is, so the blocking job can finish under its budget.
Fixes#106103.
Move the "explicit scratch means no project" decision into the one resolver
every surface funnels through, `kanban_db.create_task`: board-project
inheritance now runs only when the caller left `workspace_kind` open
(`None`), and `workspace_kind` defaults to scratch after that check. The
tool handler keeps the #106347 fix for `project=""` (no `or` collapse) and
`board=` scoping but drops its handler-local sentinel logic, since the
resolver now owns the rule; the `self_task` project inheritance for
dispatcher-owned workers is unchanged.
Sibling surfaces had the same bug through the same line and are fixed by
the same change:
- CLI `hermes kanban create --workspace scratch` on a project-scoped board
produced a project worktree; `--workspace` no longer defaults in
argparse so the resolver can tell "omitted" from "scratch".
- Dashboard `POST /tasks` with `workspace_kind: "scratch"` did the same;
`CreateTaskBody.workspace_kind` defaults to `None` for the same reason.
- `kanban_swarm.create_swarm` threads `None` through for consistency.
Tests: one resolver invariant in test_kanban_board_project.py (explicit
scratch stays scratch, omitted still inherits) and the salvaged tool test
folded into a single parametrized matrix over scoped/unscoped target boards.
Pass the caller's board into create_task and treat explicit workspace_kind=scratch or empty project as no-project so ambient board project_id cannot override MCP args.
Co-authored-by: Cursor <cursoragent@cursor.com>
Every tool.start / tool.progress / tool.complete ran three whole-timeline
passes that each allocated per part: `completeOpenStreamParts` (`map` with a
spread per row), the pending-row scan (`map` → `filter` → `map`, one wrapper
object per part), and `generatedImageEchoSources` (`flatMap` + a throwaway
array per non-image row) — then `dedupeGeneratedImageEchoesInParts` copied
the array again even when it stripped nothing, handing React a fresh
identity per event. Replace them with single in-place loops, and return the
input array from the dedupe when there are no echoes so a no-op stays a
no-op for the store.
Measured on the real `useMessageStream` hook (vitest/jsdom, 2,000 starts +
2,000 completions interleaved with text deltas): 618 ms → 138 ms; the
tool-only 2,000×2 case 252 ms → 67 ms. The remaining O(n) is one
`findIndex` per event over a ~4k array (~8 µs), well under the cost of the
copy React needs anyway, so no per-stream id index is introduced.
Behaviour is unchanged: same first-match `toolCallId` resolution, same
sparse/no-id contextual correlation, same ordering of parts.
Co-authored-by: Xipong <217837358+Xipong@users.noreply.github.com>
Health checker: keep the 12-reconnect coalesce test and replace the
stop/disconnect drop rows with the invariant #106135 names — one fresh
follow-up still runs when the active sweep fails through the handled
config-error path (13 config reads on main, 2 after the fix). Log poller:
keep single-flight + resumed cadence, and shrink the replacement-lifecycle
test to "stop suppresses a late publish and leaves no timer". Both files
were run against origin/main source and failed (11 / 13 / 13 requests).
startCompletionPoll already keeps the pane single-flight and its stop
function already suppresses a late publish, so createSerialTask (a second
queue that never forwarded its AbortSignal to getLogs) and the AbortError
re-throw inside the poll callback were defense-in-depth on top of the fix.
A profile switch now tears the old lifecycle down and starts the new one
immediately; at most one stale read can still be in flight, and its result
is discarded. Replace the AbortController with a plain stopped flag.
The salvaged fix only ran the CLI in its own session on POSIX and kept
plain subprocess.run on Windows — the one platform where the wedge in
#106244 is actually reproducible: CPython's run() retries an unbounded
communicate() after kill() there, so a grandchild holding the capture
pipes blocks the worker forever. On POSIX run() wait()s the PID and
returns promptly; the grandchild merely leaks (live-reproduced on Linux).
One code path for both platforms:
- _group_popen_kwargs: start_new_session=True on POSIX,
CREATE_NEW_PROCESS_GROUP + hide flags on Windows (replaces the
hide-only _windows_popen_kwargs).
- _kill_cli_process_group: os.killpg SIGKILL on POSIX, taskkill /T /F
on Windows (same kwarg set as the sibling taskkill sites).
- The Popen decodes with encoding="utf-8", errors="replace" like every
other subprocess call in this file (windows footgun rule).
- Drain test patches the kill helper instead of os.killpg so it runs on
every host.
Windows behaviour is not live-verifiable on this Linux host.
Drop test_gone_group_still_surfaces_timeout: the killpg-race branch is a
contextlib.suppress and the test only pinned the exact communicate()
timeout sequence (a change-detector). The real-grandchild test and the
bounded-drain test remain — the two behaviour contracts of the fix.
The group kill lives behind browser_exec's os.name != "nt" branch, so
os.killpg/SIGKILL are never reached on Windows; mark the line per the
scan's suppression convention.
subprocess.run only kills the direct CLI child on TimeoutExpired; a
browser_harness daemon / Chrome helper grandchild inherits the stdout/
stderr pipes and keeps them open, so the internal communicate() blocks
on pipe EOF forever. The wedged tool call never returns, its activity
heartbeat keeps stamping last_activity_at every 30s, and the session is
pinned at "now" in the desktop sidebar indefinitely (#106244).
On POSIX, run the CLI in its own session (start_new_session=True) and
SIGKILL the whole process group on timeout, then drain the pipes under
a bounded deadline. Windows keeps subprocess.run.
Drop the two companion tests from #106124 and #106099 that already pass
on main without their fixes (a pinned active transcript is never measured
because prune() skips it before weighing; a rerendered effect already
discards the previous closure). Keep the one red-on-main invariant per
fix: no re-serialisation of unchanged warm transcripts, and change ticks
coalescing behind an in-flight session.active_list request.
Replace the six cap-mapping unit tests (exact 5536 px value, floor, Anthropic
path, two None cases, worst-case tile math) with one behavioral test: the cap
derived from the real Codex rejection makes the production shrink pass rewrite
a real 5444x6200 PNG Responses ``input_image`` part to within the 30000-patch
budget. That is the user-visible contract; the exact pixel value is an
implementation detail. Keeps the classifier test as the second invariant.
The classifier fix routes Codex patch-budget 400s into the shrink
recovery, but _image_error_max_dimension still returned None for the
Codex wording, so the recovery fell back to the 8000 px default cap
and skipped images between ~5542 and 8000 px that already exceed the
30000-tile budget — burning the single shrink retry without shrinking
anything. Parse the reported patch limit and convert it to a per-side
pixel cap of isqrt(limit)*32 (5536 px for 30000 patches), which keeps
a square image under the budget.
OpenAI Codex Responses rejects an image whose tile-patch budget exceeds
its 30000-patch ceiling with wording ("requires N patches after
processing, exceeding the limit") that contains none of the existing
image-size vocabulary, so the 400 fell through to format_error
(non-retryable). The reactive image-shrink recovery in turn_recovery was
therefore bypassed and the session kept failing — or failover re-sent
the identical oversized image to another model.
Route the patch-budget wording to image_too_large (retryable) so the
existing shrink pass re-encodes the image under the ceiling and retries.
Fixes#106337
Drop the floor-pin test from #106290 (a snapshot of the static table); the
end-to-end merge test through provider_model_ids with the real curated floor
already fails if the slug is re-added. Note beside
_OPENCODE_FREE_KEYED_SUFFIX_MODELS why ox-alpha-free stays excluded from the
keyless catalog even though it left the curated floor.
The Go relay (GET /zen/go/v1/models) delisted ox-alpha-free 2026-09-09, but
_PROVIDER_MODELS["opencode-go"] still carried it. _profile_live_catalog merges
the curated floor into the live list (live-first for opencode-go), so the
model picker kept offering a model that now 401s — the same failure class as
#95914 (opencode-free / x-preview-f-free).
Remove it from the floor and add two regression tests: an end-to-end merge
test through provider_model_ids with the real floor (fails if a stale floor
resurrects it) and a floor-pin test asserting the known-delisted model stays
out of the offline fallback.
(cherry picked from commit 091fd85865caa09e828928f93ee985dae7e9580d)
Keep exactly the issue's Repro A (heartbeat must not become the cached answer)
and Repro B (stale READY mapping / expired button must not swallow the next
answer); both fail on origin/main and pass with the fix. The other six tests
from #106470 (config override, PENDING-tap guard, per-layer variants) were
either change-detectors or subsumed by these two.
Image and video callers were each formatting the same four-field dict into
the same sentence. Return the rendered tail from _managed_fal_billing_error
so the wording lives in one place; output is byte-identical.
Keep the two tests that fail on main without the fix:
- test_fal_common: a keyed managed submit makes exactly one POST (plus the
negative arm: an unkeyed submit still goes through the SDK retry ladder)
- test_image_generation: the 409 BILLING_ERROR body surfaces
`unsupported_pricing_meter` instead of the generic "not yet enabled" text
Dropped from #106484: the duplicate video-plugin billing test (same helper,
same assertion), the `_fal_client = fake` / `import_fal_client` stub churn
and the `tools.lazy_deps` stub — fal-client is installed in CI (`--extra fal`)
so those fixtures were not needed; the `_load_fal_client` no-op fixture on
TestManagedGatewayErrorTranslation for the same reason.
Also drop the redundant `retry_request is None` re-check in
`_ManagedFalSyncClient.submit` — `__init__` already raises when the helper
is missing.
Avoid retrying idempotent managed FAL submissions because the retry can mask the initial billing failure. Surface structured Nous billing diagnostics consistently for image and video paths, with hermetic regression coverage.
(cherry picked from commit 289ce039e9a522dc8016ae4a512214c05d0a8bc0)
finish_reason='length' has two causes: the answer was long (max_tokens reached),
or the prompt itself left no room to generate. _continue_text treated both the
same: append the fragment + a continuation nudge and retry, up to 4 times. In the
second case every retry sends a strictly longer prompt, so each attempt is worse
(Ollama n_ctx=32768: 32,638 -> 32,685 -> 32,732 prompt tokens, all truncated), the
user is told "model hit max output tokens", and max_tokens is not the lever.
The response's usage already carries prompt_tokens and the compressor already
resolves the model's context window; compare them once per truncation. Under
_MIN_CONTINUATION_HEADROOM (512) free tokens the turn ends on the first
truncation, keeps the partial text, names the context window as the cause and
points at /compress or a larger window. Unknown usage or window keeps today's
behaviour. max_tokens semantics untouched.
Co-authored-by: gaoanze888 <214786078+gaoanze888@users.noreply.github.com>
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Drop test_reap_idle_closes_doomed_and_keeps_attached from #106415: it
pins reap_idle's pre-existing selection behaviour (dead/idle reaped,
attached kept), which the fix does not change, and is not red on base.
Fold the duplicated two-idle-sessions + gated-close setup into one
helper shared by the two interleaving tests that do fail without the
fix (reap vs reap, close_all vs reap).
Teknium