The event-silence tests duplicated `_make_adapter`/`_connect` from
test_discord_liveness.py and carried a bespoke handler-wait loop. Reuse
the sibling helpers instead: `_make_adapter` grows an optional
`max_event_silence` that is only written into `extra` when given, so the
sibling's own tests keep the adapter default and stay unchanged.
Two invariants now bind the fix:
- deaf socket: a `None` stamp (no DISPATCH yet) reads healthy across
several probe intervals, and once armed, transport-green + event
silence trips `event_silence` through `_liveness_loop`.
- `websocket_event_max_silence_seconds: 0` disables only that
dimension: with a stale stamp and a stale heartbeat ACK the probe must
still run and trip on `ack_stale`. This goes red if the knob is moved
into `_start_liveness_probe`'s all-or-nothing guard (#109782).
The YAML->extra passthrough test also asserts the new key, so dropping
its `_YAML_WEBSOCKET_LIVENESS_KEYS` entry fails.
Trim the new event-silence file from 8 tests to the 2 that bind the fix:
the deaf-socket e2e through `_liveness_loop` (transport green, no
DISPATCH → probe trips with the `event_silence` reason) and the
None-stamp window reading healthy (no false trip on quiet reconnects).
The removed cases re-checked knob parsing, defaults and the YAML seed
loop already covered by the sibling liveness knobs' tests, or restated
the two kept invariants from a different angle.
The adapter reads `websocket_event_max_silence_seconds` (14400 s) but the
key was never added to DEFAULT_CONFIG, so CONFIG_SCHEMA — generated from
it — did not expose the new liveness threshold. Manual YAML worked while
dashboard users could not discover or edit it; the four sibling
websocket_* discord liveness keys are all registered there. Register it
with the same default so both config surfaces stay consistent.
Incident 2 of #109521: a Gateway socket can stay ESTABLISHED and keep
ACKing heartbeats while zero DISPATCH events are parsed, so every
transport-side liveness sample (ready/open/ack-age/latency) reads
healthy for hours. The merged #109963 deliberately dropped the
event_silence dimension: a raw-frame stamp is debug-gated
(on_socket_raw_receive needs enable_debug_events) and, since heartbeat
ACKs are frames, ack_stale always fires first by construction.
This adds the dispatch-side signal that was requested instead:
- stamp on on_socket_event_type, which discord.py 2.7.1 dispatches for
every parsed DISPATCH frame with no debug gate (verified live against
the real received_message path: 4/4 frames fired with
enable_debug_events=False, on_socket_raw_receive 0/4)
- new knob websocket_event_max_silence_seconds (default 4h, the
incident report's field-proven operator bound); 0 opts out of this
dimension ONLY — the #109782 review failure put the knob in
_start_liveness_probe's all-or-nothing guard, killing the whole
watchdog; it is gated strictly inside _read_websocket_health here
- the stamp resets per connection (connect() clears it), and a None
stamp (no event parsed yet on this connection) is not silence
- docs (en + zh-Hans) cover the new knob and the per-dimension opt-out
Fixes#109521
(cherry picked from commit b4baa97fc45794209711a45e052111d7d44d5f90)
The timeout branch of _run_hook_callback_bounded unconditionally added
gate_key to _hook_abandoned. A worker that finishes between done.wait()
returning False and the caller taking the lock has already popped its
token via _release_token, so nothing would ever clear that entry: the
callback stayed blocked for every later call id until reload with no
thread behind it. Guard the insert on the worker still being registered.
The new test makes the race deterministic by swapping the module's
threading.Event for one whose wait() lets the worker finish and then
reports a timeout, and asserts a fresh call id still runs.
Also pass tool_call_id inline from terminal_tool_result instead of the
conditional dict plumbing: an empty id is already treated as "no
identity" by _hook_call_identity and unknown fields are withheld from
narrow-signature callbacks (same shape as _fire_approval_hook). Update
the stale "(hook_name, id(cb))" comment above _hook_running_callbacks.
Hook callbacks are gated per (hook, callback, call identity). Two hooks
on the tool-loop path fired without any identity, so concurrent terminal
calls in one turn, or overlapping turns, still collapsed onto a single
gate key and the second invocation was skipped as if a callback had hung.
transform_terminal_output now forwards the tool_call_id bound in the
approval context around dispatch (only when set); transform_llm_output
forwards the turn_id already in scope. Payloads are additive: the
dispatcher withholds unknown fields from narrow-signature callbacks.
The same-call negative control fired two sequential calls with a 0.1 s
timeout, so the first call timed out and the second was dropped by the
60 s suppression window; the identity gate itself was never exercised.
Mirror the positive test instead: a 5 s timeout, two threads with the
same tool_call_id while the callback is held on an Event.
Add one test for the abandoned-worker gate: a hung callback followed by
a call with a fresh tool_call_id, with suppression zeroed, must start
exactly one worker. Reverting the gate change makes it fail.
Gating hook callbacks by call identity lets two concurrent calls of the
same tool both run their hooks, but it also let a fresh tool_call_id pass
the gate once the 60 s suppression window lapsed even though the previous
worker for that callback never returned. A hung plugin then leaked one
daemon thread per minute for the life of the process; on the old
coarse-keyed gate it leaked exactly one.
Track abandoned-but-running workers per callback: the timeout branch
records the gate key, the worker's own release discards it, and the gate
treats any non-empty abandoned set as "still running" for that callback.
Healthy callbacks keep distinct-id concurrency; hung ones are back to
at most one outstanding worker.
Concurrent invocations of the same tool in one session collapsed into a single
busy key (hook_name, id(cb)): the second invocation was reported as 'still
running' and dropped. For pre_tool_call a drop is a fail-closed block, so the
gate silenced itself on an ordinary, healthy callback.
Measured on a busy profile: 3574 skip lines and 0 timeout lines in one hour —
every skip was the 'while still running' branch, i.e. pure key collision, not
slowness.
The gate now keys on the call identity that is already in the payload
(tool_call_id, else turn_id, else none — the last case behaves exactly as
before). Suppression stays keyed coarsely on (hook_name, id(cb)): a hung
callback is a fact about the callback, so its back-off must not be diluted
per call.
Refs #98382. Independent of #107894 (that one releases the slot on timeout;
this one stops healthy concurrency from colliding).
(cherry picked from commit 53b3dacd008418fcdf5fa6dfcadde575a35a776e)
`_fd_is_truly_unlinked` stats `/proc/<pid>/fd/<n>` after the scan has
already read the readlink target. Between those two steps the descriptor
can be closed (ENOENT, handled by the previous commit) or the whole
process can exit (ESRCH). Both mean the descriptor can no longer keep a
retired WAL/SHM generation alive, so neither is evidence of a live holder
and neither should make the guard refuse to open the database.
Match the sibling scan in hermes_state_holders, which already skips both
errnos, by branching on `exc.errno in (ENOENT, ESRCH)`; any other OSError
still fails closed. The existing closed-descriptor test is parametrized
over both errnos, injecting the failure at `os.stat` so the ESRCH path is
exercised on every platform.
`\bsocat\b` under IGNORECASE matched "SOCAT", the Surface Ocean CO2 Atlas,
in every oceanography skill of a 2,110-file research bundle (17 critical
findings in one file), burying the bundle's real issues under noise. A real
socat relay always names an address type (TCP:/UDP:/OPENSSL:/EXEC:/SYSTEM:/
PTY:/UNIX-…:), so the pattern now requires one on the same line. `nc -l` /
`ncat -l` are unchanged. Scanner version bumped to v5 so cached verdicts
re-scan.
Catalog entries sort official → stars desc → name, both in browse shelves and
filtered grids, with a ★ pill on each card linking to the repo's stargazers.
Rate-limit discipline is the design constraint: the docs site deploys many
times a day and shares one GitHub App API budget with every other workflow
(tonight's merge train got rate-limited on unrelated uploads). So
website/scripts/fetch-plugin-stars.py first fetches the live site's own
plugin-stars.json (a CDN GET, not the API); if that cache is under 24h old it
is reused verbatim and GitHub is never called. Only a stale cache triggers one
GET /repos/{owner}/{repo} per unique catalog repo, and a 403/429 mid-run keeps
the previous counts instead of zeroing them. extract-plugins.py merges the
cache into plugins.json (`stars`) and plugins-meta.json (`starsFetchedAt`), and
the page footnote says when the ranking was last refreshed.
Tab-to-grill a draft in the Desktop composer: one high-leverage decision per rung with a
recommended answer, then a faithful execution brief placed in the composer for review.
Owner-submitted; no self-updater; empty capability block matches register() at the pin.
Teknium's call: most community submissions are Desktop panes, so an entry
without a category lands on the Desktop shelf; "other" becomes "general" for
plugins that genuinely span areas. Shelf order puts Desktop first. The six
entries merged today (pets-all, newswire, auto-titler, live-voice,
metamask-wallet, web-octen) get explicit categories.
The catalog page was one undifferentiated grid filtered only by tier, so a
memory provider sat between two Desktop panes. Entries now carry an optional
``category`` (memory | desktop | platform | web | tools | voice | automation |
models | other, default other) that the loader, the admission validator and
the site extractor all understand.
/docs/plugins renders one shelf per category in browse mode, a category pill
row under the tier pills, a clickable category chip on every card, and a
results bar (active category, count, clear) when a filter or search flattens
the view. ``hermes plugins catalog`` gains a Category column and groups by it.
All 18 shipped entries are categorised. Unknown categories fail admission
(same contract as tier) so a typo cannot create a phantom shelf.
`ProcessRegistry._move_to_finished` pops the session out of `_running`, then
saves the receipt, releases handles and writes the checkpoint, and only THEN
enqueues the completion and sets `_completion_event`. A quiet one-shot parent
whose turn ends inside that window called `wait_for_pending_completions`,
found nothing in `_running`, drained an empty queue and exited without the
follow-up turn. That is the CI flake in
tests/tools/test_completed_process_results.py::test_headless_terminal_result_survives_cli_exit
(`follow_ups == []`), which also hit unrelated branches.
Consider `_finished` sessions whose event is not yet set as pending too.
Repro: a 6s sleep before the enqueue plus a 2s delay before the parent's first
wait fails the E2E 2/2 on main and passes 2/2 with this change.
osv-scanner.yml documents itself as detection-only (fail-on-vuln: false,
findings land in the Security tab) yet all-checks-pass listed it in needs,
so any failure result blocked the merge. In practice the failures are not
vulnerabilities: the "Upload to code-scanning" step hits GitHub's
per-installation API rate limit whenever several PRs run at once, and a
merge train of catalog entries went red on it across the board. The scan
still runs on every PR and weekly on main; it just reports instead of gating.
Dropping a link out of a browser onto the Desktop composer toasted
"Drop files — Could not attach <title>.url" and attached nothing. A browser
link drag carries `text/uri-list` plus, on Windows, a virtual `<title>.url`
shortcut File (`.webloc` on macOS) that has no on-disk path. The drop
pipeline only understood Files and in-app paths: the path-less stub went to
the upload branch, `attachContextFilePath('')` returned false, and the user
had to copy/paste the URL instead.
`extractDroppedFiles` now reads `text/uri-list`, drops the path-less
shortcut stub when a link is present, and emits `{ url }` entries;
`droppedFileInlineRef` turns them into the same `@url:` chip the "+ → Add
URL" dialog and paste-linkify produce, so every drop surface (composer
form, text box, conversation area, edit composer) gets it for free.
`dragHasAttachments` accepts `text/uri-list` so the form-level enter/over
handlers claim the drag at all. A path-less *image* dragged off a web page
keeps its bytes and wins over the link to its own src.
Attribute drive-level errors to the thread being drained, not the send
that created the queue. Reinsert repeat member failures in recency order
so the collapsed activity row cannot show an older sibling failure.
Cover both invariants and the repeated-refusal sequence in native Desktop.
Thanks to @kvnloo for identifying both review findings.
kshitijk4poor