Commit a270c4ade's session-key fallback in _resolve_container_task_id was
added to stop cross-profile SSH environment reuse, but it wasn't backend-
gated: persistent Docker silently fragmented into one container per gateway
session, breaking the product contract (one long-lived container per profile,
shared by CLI and every session of that profile). #93950's vanishing MEDIA
attachments were downstream damage.
- persistent Docker (container_persistent: true) now keys to the profile:
literal 'default' for the default profile (same container as CLI),
'profile:<name>' for named profiles
- SSH and non-persistent Docker keep session scoping (the original leak fix
and the #82731 isolation contract are untouched)
- gateway MEDIA translation follows the profile layout and keeps the legacy
bug-window per-session sandboxes as fallback candidates, trying each until
the file resolves — old sessions self-heal, no migration
- /root/.hermes credential-surface refusal preserved across all layouts
Phase 2c on the full final diff flagged the re-export shim as
contradicting the adjacent house pattern (agent.replay_cleanup import,
which documents retiring private aliases once tests migrate). Migrate
all six tests to the canonical gateway.media_repair seam, import the
canonical name in run.py, and drop the dead 'and result' guard at the
background-task call site.
- Make repair_explicit_computer_use_media_paths fail-open internally
(cosmetic repair must never abort delivery); drop the cron-only
try/except so all three call sites are identical one-liners.
- Drop cron's redundant 'MEDIA:' pre-check (helper early-returns).
- Document the intentional lazy BasePlatformAdapter import (verified:
no cycle either way; keeps module import cheap for cron processes).
- Point the two new regression tests at the canonical
gateway.media_repair seam; pre-existing tests keep pinning the
gateway.run re-export shim.
- Docstring: matching is case-insensitive, say so.
Follow-up to the salvaged fix from PR #94439:
- Extract the repair into gateway/media_repair.py (shared module) and
re-export under the historical private name in gateway/run.py.
- Wire the repair into the two bypassed delivery surfaces: gateway
background tasks (_run_background_task_inner) and cron job delivery
(cron/scheduler.py) — both call agent.run_conversation directly and
never pass the main turn chokepoint.
- Fail closed on malformed/truncated JSON tool results: parse JSON-looking
content first instead of regex-scanning the raw string, which yielded a
doubled-backslash path artifact and rewrote the response to a path the
model never wrote.
- Deduplicate the tool_name_by_call_id builder (three verbatim copies in
gateway/run.py) into the shared module; hoist the abs-path prefix regex.
- Add regression tests: malformed-JSON fail-closed (mutation-checked) and
the compression-fallback last-user slice (incl. no-user fail-closed).
Same follow-through as the other filter-static stub updates: the three
lambdas patching filter_local_delivery_paths rejected the new keyword
and failed CI (tests/gateway/test_73771_media_resend_dedup.py).
De-silence the #93950 failure mode: when a container-absolute MEDIA path
under /workspace or /root cannot be resolved to a host sandbox file while
TERMINAL_ENV=docker, log the reason (no mounts / no prefix match / host
file missing) plus the delivering session key instead of only the generic
'Skipping unsafe MEDIA directive path' line upstream.
Persistent Docker containers bind <sandboxes>/docker/<task>/{workspace,home}
where <task> is sanitize_task_id_for_path("session:<session_key>") — but the
gateway's synthetic mounts hardcoded the literal "default" sandbox
(_default_docker_workspace_host_root / _docker_persistent_home_host_root).
For any session-scoped deployment the longest-prefix match missed, the
container path fell through to a host-filesystem resolve that could not
exist, and every MEDIA attachment was silently dropped.
The post-handler delivery pipeline also runs after
_handle_message_with_agent cleared the turn's session contextvars, so even
a correct sandbox derivation consulting ambient state would collapse onto
"default". Thread the delivering session's key explicitly through
validate_media_delivery_path -> _translate_docker_container_media_path ->
the two host-root helpers (same pattern as the TTS fix for #57049/#36685).
Default-sandbox resolution and the /root/.hermes credential exclusion are
preserved; contexts without a key keep the historical behavior.
Pin the NoneType crash: TEAMS_SDK_AVAILABLE true plus a failed bind must
return False without calling App(). Also pin plugin import when the
microsoft_teams parent namespace is missing.
The adapter's private thread-deadline helper was the ancestor of the
unified deadline layer's run_bounded_async (#85147 was extracted from
it, plus the caller-cancellation leak fix the original still lacked).
Consolidate: the helper body becomes a thin wrapper mapping
BoundedResult.timed_out back to the asyncio.TimeoutError its 9 call
sites (the PTB retry ladder) expect. ~90 duplicated lines die, along
with the adapter-local copies of the abandon-cleanup runner and the
blocked-loop faulthandler diagnostics (both live in agent/deadline.py).
Everything the call sites rely on is preserved by the unified layer:
- thread-timer deadline that survives a blocked event loop (#63309)
- abandonment of cancellation-shielded tasks (PTB/httpcore anyio init)
- detached best-effort on_abandon cleanup (no httpx pool leak per retry)
- off-loop stack dump when the loop never processes the expiry
Plus one behavior IMPROVEMENT inherited from the shared copy: a caller
cancelling the wrapper no longer leaks the inner task unobserved (the
telegram original had that leak; the extraction fixed it).
test_telegram_init_deadline.py: the #63309 diagnostics probe now pins
the shared layer's dump hook (label "telegram-init") — same contract,
new seam. Wedge + cleanup-crash tests pass unchanged.
PairingStore(profile=None) resolved its storage directory from the
module-level PAIRING_DIR constant, which was computed exactly once, at
module import time. A long-lived process (the gateway, started once at
container/process boot) can import this module before HERMES_HOME or a
profile's context is fully established, freezing PAIRING_DIR to a wrong
value for the rest of that process's lifetime -- even though a freshly
started, short-lived process (e.g. the `hermes pairing` CLI) re-imports
the module later with the environment already correct.
That asymmetry is exactly what made pending pairing codes issued by the
gateway process unrecoverable (the pending-code write landed under the
stale, wrong directory) while CLI-invoked writes to the same nominal
directory kept working -- see #93449 for the full writeup and a live
reproduction. tests/hermes_cli/test_dashboard_admin_endpoints.py already
carried a comment acknowledging this exact staleness in passing ("the
module-level PAIRING_DIR is bound at import"), and
TestProfileScopedStorage::test_default_store_uses_global_dir's own
comment describes working around it rather than it being intentional
behavior -- this fixes the underlying cause both were compensating for.
The profile-scoped branch already resolved its directory lazily inside
__init__ (matching this docstring's claim that resolution is lazy); this
brings the non-profile branch in line with it.
Fix keeps PAIRING_DIR as the same test seam already used throughout the
test suite (`patch("gateway.pairing.PAIRING_DIR", tmp_path)`, ~30 call
sites) unchanged: it's now a None sentinel instead of an eagerly computed
path, and a new _default_pairing_dir() helper resolves it fresh on every
call, honoring a patched (non-None) value when one is set. No existing
test needed to change.
Added a regression test that does not patch PAIRING_DIR directly and
instead exercises the real lazy-resolution path across two different
HERMES_HOME values in the same process -- confirmed it fails on the
pre-fix code (gets stuck with whatever the first PairingStore() call in
the test session happened to see) and passes with the fix.
Verified: tests/gateway/test_pairing.py (39, incl. the new one),
tests/hermes_cli/test_pairing.py, and tests/tools/test_pr_6656_regressions.py
all pass unmodified.
Under gateway.multiplex_profiles, secondary profiles are constructed
inside _profile_runtime_scope and their .env lives in the profile's
secret scope - gateway/run.py explicitly does NOT mutate os.environ with
it. Four adapters still read their AUTHORIZATION config via raw
os.getenv, so every secondary profile either (a) silently missed its own
env-only allowlists/policies (fail-closed: all DMs dropped at intake) or
(b) inherited the default profile's GATEWAY_ALLOW_ALL_USERS=true /
allowlists from the shared process env (fail-open admissions):
- weixin.py: WEIXIN_DM_POLICY / WEIXIN_ALLOWED_USERS /
WEIXIN_GROUP_ALLOWED_USERS / WEIXIN_ALLOW_ALL_USERS +
GATEWAY_ALLOW_ALL_USERS in _open_dm_opted_in
- yuanbao.py: YUANBAO_DM_POLICY / DM_ALLOW_FROM / GROUP_POLICY /
GROUP_ALLOW_FROM / ALLOW_ALL_USERS (new _yb_secret helper; AccessPolicy
hard-gates intake)
- signal.py: SIGNAL_GROUP_ALLOWED_USERS / SIGNAL_ALLOWED_USERS (new
_sig_secret helper; empty scoped group list previously meant "drop all
groups" silently)
- wecom/adapter.py: WECOM_DM_POLICY / WECOM_ALLOWED_USERS /
WECOM_GROUP_POLICY / WECOM_ALLOW_ALL_USERS + GATEWAY_ALLOW_ALL_USERS -
while credentials one line above already used _get_scoped_secret
- gateway/run.py::_own_policy_open_startup_violation: the open-policy
startup guard validated GATEWAY_ALLOW_ALL_USERS via raw os.getenv even
though its sibling dm/group reads already used the scoped _getenv
All reads now go through the canonical fail-closed scoped shape QQ's
_resolve_qq_secret already used (scope hit wins; unscoped single-profile
callers keep legacy os.environ behavior). Regression suite drives the
real scope contextvar across all four helpers plus the admission gates
and the startup guard, asserting both directions: profile values are
visible under multiplex, default-profile values never leak.
Fixes#93522
WEIXIN_DM_POLICY/ALLOWED_USERS/GROUP_ALLOWED_USERS, YUANBAO's equivalents,
WECOM_DM_POLICY/ALLOWED_USERS/GROUP_POLICY, and the startup guard's
GATEWAY_ALLOW_ALL_USERS check still read raw os.getenv at adapter
construction time. Under gateway.multiplex_profiles that reads the process
env instead of the per-profile secret scope, so a secondary profile either
silently drops every DM (its own env-only allowlist is invisible) or
inherits the default profile's allow-all/allowlist config.
Route these reads through the existing scoped helpers (_wx_secret,
_get_scoped_secret, gateway.authz_mixin._platform_gate_env, and
gateway.config._getenv) already used for the adjacent credential reads in
the same adapters.
Fixes#93522.
Review v2 of #93084: the readable-cmdline path used substring matching
as destructive authority. That fails open on prefix collisions
(--profile timothy vs our profile tim) and same-name profiles under
different roots, so a poisoned record could still reach SIGTERM.
Ownership is now decided by the persisted identity record ALONE — exact
_same_hermes_home equality, bound to the live target by exact pid +
start-time. Missing/legacy/unbound/foreign records all refuse. A
readable argv feeds only a token-exact consistency check
(_looks_like_profile_conflict_from_cmdline via shlex tokens) that
refuses explicit contradictions like --profile timothy under tim; bare
or matching argv adds nothing.
Also fixes the review's source-of-record concern: the guard validates
the record that authorizes get_running_pid()'s answer rather than
assuming {HERMES_HOME}/gateway.pid is always the source.
Adds the requested signal-boundary regression: start_gateway(replace=
True) with unprovable ownership returns False without calling
terminate_pid or writing a takeover marker; the bound same-home
counterpart still reaches the replace flow. Legacy replace-flow tests
updated to stage a valid bound record for their legitimate-replace
fixtures.
_auth_env fell through to os.environ on a scoped miss, so one profile
could inherit another profile's allowlists and allow-all flags.
bot_relay.waiter_command put connection_id into python -c source. A
quote in the id broke the waiter. A crafted id could run extra Python
in the sender gateway.
/review takes the last 10 chat messages plus optional instructions,
spawns a full-privilege background subagent (the async delegation
rail) that investigates the referenced work (PR, code, docs), and its
complete review re-enters the spawning session as a normal
async-delegation completion the primary agent can act on.
- agent/review_engine.py: shared engine (snapshot, briefing,
auxiliary.review credential resolution, dispatch, note formatting)
- tools/delegate_tool.py: internal credentials_cfg per-call override
(never model-facing) resolved through the same credential system as
delegation.provider pins
- auxiliary.review config block (provider/model/base_url/api_key/
api_mode); provider auto + empty model = inherit the main model
- Surfaces: CLI process_command, gateway run.py dispatch +
slash_commands handler (binds the approval session key so the
completion routes back), TUI/Desktop live dispatch in
tui_gateway/server.py, CommandDef registry (+Slack /hermes-only cap)
- Docs: delegation.md section + slash-commands.md (both tables)
- Tests: 15 engine tests (sabotage-verified: credentials_cfg and
dispatch tests fail without the fix), 4 gateway handler tests
through the real async rail
A /p/<profile>/ URL prefix on a gateway with multiplex_profiles off was
silently ignored: the request was handled as the gateway-owning profile,
so /p/lokaj/v1/toolsets reported the OWNER's platform_toolsets (and every
other profile-owned config read — skills, capabilities, model options,
agent-run toolset resolution — resolved from the owner too). That is the
exact repro in #91583 defect 2: enabling computer_use with
'hermes -p lokaj tools enable computer_use --platform api_server' showed
enabled in lokaj's config while /p/lokaj/v1/toolsets stayed false, and
enabling it on the owner profile flipped it true.
Per-profile capability isolation is the intended design (ruling on
a different profile's config. Multiplexed gateways were already correct —
the profile-prefix middleware enters _profile_runtime_scope and every
canonical config loader honors the HERMES_HOME override contextvar
(verified empirically for load_config, get_config_path and
_load_gateway_config) — the leak was only the non-multiplex fallthrough.
Fix at the one seam both adapters share: _resolve_request_profile now
rejects (404) a prefix naming any profile other than the one the gateway
actually serves. A self-referential prefix (/p/default/ on the default
gateway, /p/lokaj/ on a gateway launched for lokaj) still falls through
so existing well-formed clients keep working. Same change in the webhook
adapter, which had the identical fallthrough. New shared helper
hermes_cli.profiles.profile_matches_home does the home comparison,
fail-closed.
Tests: tests/gateway/test_multiplex_toolsets_profile_isolation.py —
E2E-style with two real profile homes + config.yamls under a temp
HERMES_HOME, real aiohttp routing through the profile-prefix middleware:
per-profile /p/<x>/v1/toolsets isolation for both owner and secondary
(the #91583 repro asserts computer_use true under /p/lokaj only),
cross-profile key rejection, and the fail-closed non-multiplex prefix
for both adapters. Sabotage-verified: reverting the adapter change fails
the 3 fail-closed tests.
Fixes#91583 (defect 2). Repro and live validation by @kubaboski.
The prefix is an address: the caller is naming WHICH agent the request is
for. With gateway.multiplex_profiles off, _resolve_request_profile ignored
the prefix entirely — "don't 404 a would-be valid route" — so a request
explicitly addressed to one agent was silently answered by a different one.
Observed live (Aug 2026): `hermes peer dm mini/researcher` was answered by
the mini's DEFAULT agent with no error on either side, because that host
runs one LaunchDaemon per profile and only the default daemon hosted an
api_server. A wrong-agent answer is strictly worse than an error: the
sender believes the addressee got the message.
With multiplexing off the process serves exactly one profile, so the prefix
is honored when it names that profile (peers address single-profile daemons
this way without knowing the host's topology — get_active_profile_name() is
the same identity the file already uses for model resolution) and rejected
otherwise through the existing _PROFILE_REJECTED path (404). A process that
cannot resolve its own identity rejects too: if it cannot prove who it is,
it must not answer as anyone.
Unprefixed requests are untouched, and multiplexed hosts are untouched —
the change is confined to the prefix-present, multiplexing-off branch that
previously discarded the caller's addressing.
Bot Mode always hides canonical 'Bot Chat' sessions, but _find_bot_chat's
GET /api/sessions listing used the default include_hidden=False path, so
the existing hidden row was invisible, _ensure_bot_chat tried to create a
duplicate, and the peer DB's UNIQUE(title) guard rejected it — DM failed.
- api_server: GET /api/sessions now accepts an exact-title lookup
(?title=...) and honors include_hidden=1 ONLY alongside a title filter,
so canonical hidden rows resolve without exposing a blanket hidden
listing on the client surface. The title needle is pushed into SQL
(search_query) so old hidden rows outside the recency window are found.
- peer dm client: _find_bot_chat sends title + include_hidden=1; older
peers ignore the unknown params and degrade to today's behavior.
- Clear diagnosable error on the older-peer duplicate-create rejection,
naming the hidden canonical chat and the PATCH hidden:false workaround.
- Unit tests (hidden resolution, no duplicate create, older-peer error,
older-peer visible fallback) + real-gateway E2E over a real state.db.
Root-cause analysis and regression recipe by @kubaboski in #91583.
Fixes#91583
- bind under umask 0o177 so the socket is never world-connectable, even
pre-chmod (review pt 3)
- verb handlers run in an executor: state-file reads stay off the
adapter event loop (pt 2)
- inventory dedupes one multiplex gateway answering identify for several
homes — one runtime record per pid, with regression test (pt 1)
- v1 wire contract (one request per connection) documented in the module
docstring (pt 5); /tmp-unwritable skip in the short-home test
Live-verified: perms 600 at bind, identify 4.3ms via executor path, 20
rapid queries healthy.
First windows-latest run proved the design premise in miniature: identify
answered pid 616 while Popen.pid said 8000 — uv's Windows python.exe is a
trampoline that spawns the real interpreter as a child, so the spawner's
PID view is wrong and the process's self-declaration is right. Assert
against the child's printed os.getpid(); use taskkill /T for teardown.
CI runners put pytest tmp roots past sun_path, which routed every test
home through the fallback: two tests assumed in-home binding. Tests now
assert against the resolved bind location, and the fallback itself
prefers /tmp when tempfile.gettempdir() is too deep to fit sun_path.
Real child process binding the real pipe via the proactor loop with the
default handlers; real sync client; real collect_fleet_versions consumer;
kill-and-fallback proof. Skipped everywhere except a real Windows host.
The gateway now creates a local control socket at startup (Unix domain
socket at $HERMES_HOME/gateway.sock with a pointer-file fallback for
long paths; named pipe on Windows) and answers versioned JSON verbs:
- identify: pid, profile, hermes_home, code_sha/code_version (#91283
stamps, now queryable live), self-declared supervisor kind, start_time
- status: the live runtime-status payload, answered by the process itself
Bound immediately after the PID-file O_EXCL claim (the moment the
process becomes the authoritative gateway for its HERMES_HOME), removed
on clean shutdown; a successor clears any stale socket on bind. Strictly
non-fatal: bind failure only means consumers use the old path.
Consumers migrated (observability only, scan layer demoted to fallback,
never deleted):
- collect_fleet_versions() (post-update fleet matrix): prefers a live
identify answer over gateway_state.json; entries carry source=socket
- collect_runtime_inventory() (hermes update --plan): prefers the
socket, and takes the gateway's own supervisor declaration instead of
inferring it from PID scans
Old gateways mid-upgrade, crashed processes, and bind failures behave
exactly as before. Never a TCP port; filesystem/pipe ACLs are the auth
boundary (0600 socket).
Part of #91277 (fleet-update reliability). Design: #92091.
Addresses both review findings from @egilewski on #89134:
- Non-finite values: _coerce_int now degrades int(inf) (OverflowError
previously ABORTED gateway config loading); the clamp requires
math.isfinite plus sane upper bounds (interval <=3600s, timeout
<=600s, strikes <=1000), falling back to the shutdown_watchdog
constants.
- Loader wiring: load_gateway_config builds gw_data FLAT and never
forwarded the yaml gateway: section, so loop_watchdog* keys —
including the PRE-EXISTING loop_watchdog bool documented in
config_defaults — were silently ignored on the real startup path.
Bridged with the established top-level-wins/nested-fallback pattern.
E2E: config.yaml with loop_watchdog:false + strikes:12 + interval:.inf
now yields False/12/30.0 through the real loader.
Downscope of the salvaged #89134 per review: the 3->8 default raise was
symptom tolerance for the false-positive class the off-loop heartbeat +
two-witness probe fixes at the root — fleet-wide it would only delay
genuine-wedge recovery ~2.7x. The three tuning knobs keep independent
operator value and stay:
- default max_strikes back to 3 everywhere (constant, dataclass,
from_dict fallback, floor clamp, tests)
- gateway/config.py + gateway/run.py now reference the
shutdown_watchdog DEFAULT_* constants instead of duplicating literals
in three places (drift hazard)
- knobs registered in hermes_cli/config_defaults.py alongside the
sibling gateway.loop_watchdog bool
The event-loop liveness watchdog (gateway.shutdown_watchdog) hard-exited with
code 75 after 3 consecutive missed probes (probe_interval=30s, timeout=10s,
max_strikes=3), i.e. ~90-120s of loop block. Telegram/Discord reconnect during
a network blip does synchronous socket I/O on the loop and can block it for
60-90s; these stalls self-recover (recurring fleet incidents on 2026-08-17
stalled cron dispatch ~21h via restart churn, kanban t_0f76430f).
Raise the default max_strikes 3->8 so a transient reconnect stall is tolerated
while a genuine multi-minute wedge still escalates, and expose the three
tolerance knobs via config.yaml (gateway.loop_watchdog_probe_interval_s /
_probe_timeout_s / _max_strikes) so operators can tune per deployment.
Refs: kanban t_70483f23
Post-merge follow-up to #92173. The claim + resume-clear lived inside
the boot-send task AFTER the restart notification — itself a
flood-controllable send. If that notification outlived the restore-gate
timeout, the gate opened with zero rows claimed and the resume
scheduler replayed turns whose answers were already in the ledger,
while the background task later redelivered them too (duplicate
delivery + re-paid turn).
Split _redeliver_pending_obligations into _claim_pending_obligations
(pure DB: sweep + resume clear, awaited inline before the send task
exists) and _redeliver_claimed_obligations (network half, stays inside
the bounded task). The original name remains as a composition wrapper.
Mutation-checked: both updated gate tests fail on pre-split run.py.
Spawn-time Context isolation cannot rewrite an already-running watcher task. Run dispatcher SQLite offloads in an empty Context so write_txn no longer false-trips after delegate_task, while real child callers still hit the mutation guard.
Review of #90448 by @andrexibiza: adding _ensure_reconnect_watcher_running()
to the already-queued branch of a fatal callback is still an event-coupled
check. It needs a later fatal error from some other platform to arrive, and
#81036 makes that less likely rather than more -- it publishes the queue
before disconnect and drops the failed adapter from the live map, so after
the watcher's supervised restart budget is spent there may be no adapter
left to emit the event recovery is waiting on.
That is the state #72366 (salvage of #71867 by @ygd58) restored supervision
to close: queued work exists, the watcher is dead, and nobody owns the
invariant. Supervision being finite is correct; having no owner past the
budget is not.
_spawn_supervised now takes on_give_up, invoked when it abandons a task --
the supervisor is the only thing that knows it has. The reconnect watcher
uses it to hold:
while _running and _failed_platforms is non-empty, either a reconnect
watcher is live or a bounded respawn is scheduled.
Empty queue: leave it down and log; the enqueue path spawns a fresh watcher
the moment something depends on one. Non-empty: a bounded slow tier at
_RECONNECT_WATCHER_SLOW_RETRY_SECS (300s) for _MAX_SLOW_WATCHER_RESPAWNS (6)
attempts, standing down early if the queue drains or a watcher returns on
its own. Exhausted: one loud error naming the platforms left unattended.
The ceiling is (1 + _MAX_SUPERVISED_RESTARTS) x (1 + _MAX_SLOW_WATCHER_RESPAWNS)
spawns -- 42 across at least half an hour -- because each slow attempt hands
the watcher a fresh supervised budget. A test asserts that ceiling so it
cannot quietly become a restart loop.
Deliberately NOT included: requesting a process restart when the slow tier
is also exhausted. Taking down every healthy platform to heal a sick one is
a blast-radius policy decision for a maintainer.
Two things this turned up:
- _spawn_supervised did not thread on_give_up through its own backoff
respawn, so the callback was lost after the first restart and the give-up
branch had no owner at exactly the moment it needed one -- the same defect
the on_spawn docstring warns about, one parameter over.
- Three call sites repeated the (factory, name, on_spawn) triple, whose
on_spawn half is load-bearing. They now go through
_spawn_reconnect_watcher().
_supervised_backoff() names the previously-inline exponential schedule so
the exhaustion tests can collapse it; production behaviour is unchanged.
Refs #90386
_ensure_reconnect_watcher_running() exists for one situation: the reconnect
watcher has exhausted _MAX_SUPERVISED_RESTARTS, so _spawn_supervised has logged
"giving up restarts" and will never bring it back on its own (#70344, and the
supervised-restart half of #71758). It had exactly one call site, inside the
newly-queued branch of _queue_retryable_fatal_platform.
That branch is unreachable for a platform already in _failed_platforms, which
is the only kind of platform the watcher can have been retrying long enough to
burn five rapid restarts on. So the backstop could not fire in the one state it
was written for.
The failure is silent by construction. The early return logs nothing, so there
is no "queued for background reconnection" line. The stranded check in
_handle_adapter_fatal_error_detached deliberately treats a queued platform as
safe, so the gateway does not exit for the service manager either. With another
platform still connected, self.adapters is non-empty and the "gateway staying
alive, watcher will retry in background" branch is skipped too. A retryable
fatal error can therefore produce a single ERROR line and then nothing: the
platform sits in the queue that nobody is draining until someone restarts the
process by hand (#90386 reports 4h17m of that, with cron unaffected throughout).
Call the ensure on the already-queued path as well. It is already idempotent
and already cheap: it returns immediately unless the tracked task is done, and
it routes through the same on_spawn handle tracking, so a live watcher is never
duplicated.
The queue entry itself is deliberately left untouched. Re-enqueueing would
reset attempts and next_retry, restarting the backoff ladder on every fatal
error and hammering a provider that is already refusing the connection.
Restart notification and obligation redelivery ran before the
startup-restore gate opened, so one hung Telegram send queued inbound
on every platform. Bound those sends with the same timeout the resume
gate already uses, and clear resume_pending before send so a timed-out
redelivery cannot also replay the turn.
Telegram RetryAfter on send() slept the server retry_after with no
ceiling, so a 97-minute penalty pinned the coroutine. Mirror the edit
path: waits over 5s return immediately; short waits still retry inline.
The Discord /model picker built a single discord.ui.Select filled with
models[:25], silently dropping any models beyond the first 25. Discord
caps a single select at 25 options but allows up to 5 component rows, so
partition the list across up to 3 select menus (25 each; Back/Cancel use
the other 2) instead of truncating.
This fixes providers like Nous (curated list + Portal recommendations
exceed 25) whose tail — including free-tier :free Portal picks — was
previously clipped on Discord while showing fine in the Portal UI / CLI.
- _build_model_select: slice into <=25-option chunks, one select per chunk
(custom_id model_model_select_<i>), all via _on_model_selected. Multi-row
menus get a (n/total) placeholder suffix.
- _on_provider_selected: 'N more available' count reflects models actually
rendered across the partitioned menus.
- Add regression test covering the 37-model Nous case (no truncation/dupes,
per-menu 25 cap holds).
Teknium