Commit Graph

2250 Commits

Author SHA1 Message Date
Teknium fc46eabcf5 refactor(plugins): AST-neutral layout compaction of loader, chronos, disk-cleanup, google_meet 2026-09-02 20:47:48 -07:00
Teknium d9beaff938 refactor(langfuse): drop orphaned _capture/_trace modules (never imported; superseded by __init__) 2026-09-02 20:34:44 -07:00
Teknium 4d1880e0bf fix(integration): restore subprocess encoding/stdin guards dropped in simplification
Simplification workers collapsed subprocess call sites into shared kwargs
helpers and dropped the Windows/TUI safety kwargs on the way:

- encoding='utf-8', errors='replace' restored on text=True runs in
  copilot_acp_client, hermes_cli/setup (vercel install), managed_uv
  (codesign steps), local_runtime/hardware._stdout, a2a adapter.
- stdin=subprocess.DEVNULL restored on copilot probe, verify/runner
  _SUBPROCESS_KW, iron_proxy._run, google_meet playwright/system_profiler,
  simplex convert, whatsapp _RUN_TEXT, mem0 ollama serve Popen.
  google_meet sudo/brew install keeps inherited stdin (user-confirmed,
  may prompt) — marked noqa: subprocess-stdin.
- Windows-safe SIGKILL: getattr(signal, 'SIGKILL', SIGTERM) in
  verify/runner; photon _kill call re-marked windows-footgun: ok
  (unreachable on win32).
- scripts/check_subprocess_stdin.py now recognizes **kwargs splats
  (**_KW / **_kw(...)) ONLY when the same-file definition provably sets
  stdin= — covers tui_gateway _capture_run_kwargs/run_kw. Parity test added.
2026-09-02 16:36:10 -07:00
Teknium ea6644e1dc fix(integration): telegram dm-topic persist reads config.yaml via read_user_config_raw()
Raw yaml.safe_load of config.yaml is only legal in owner modules; the
write-back round-trip uses the sanctioned raw primitive. Behavior-identical
(same file, same parse, {} on missing).
2026-09-02 15:41:16 -07:00
kshitijk4poor 60ee3790e0 fix(gateway): publish degraded status when polling dies mid-session, via a public seam
Follow-ups on the #101406 salvage (#101391):

- The issue's measured incident was a healthy connect followed by silent
  polling loss; nothing republished for 11 h. `_schedule_polling_recovery`
  (the single entry to the recovery ladder) now publishes "retrying" while
  the adapter is running; `_record_polling_progress` already flips it back.
- `BasePlatformAdapter.send_path_degraded` property (default False) replaces
  the runner's `getattr(adapter, "_send_path_degraded")` reach into a
  plugin-private attribute; Telegram overrides it. `_mark_connected()` reads
  the property instead of taking a `degraded=` kwarg, and `_mark_degraded()`
  + `DEGRADED_STATUS_MESSAGE` replace three copies of the same write/string.
- The startup connect stamp (sibling of the reconnect stamp) honours the
  same flag.
- Tests: mid-session death publishes retrying; pre-connect death does not;
  progress never flips while not running / fatal; runner reconnect stamp
  honours the adapter flag.
2026-09-03 03:47:38 +05:30
chelsealong ce52ae21a7 fix(telegram): do not publish connected for a degraded, unconfirmed polling path
connect() intentionally returns True when Telegram polling starts in
degraded mode (or a reconnect's require_progress=False skips the strict
readiness gate), so background recovery can retry without failing
gateway startup. But _mark_connected() published platform_state
"connected" unconditionally, and the reconnect watcher in gateway/run.py
stamped "connected" again right after -- so gateway_state.json was
indistinguishable from a healthy adapter for as long as recovery took
(observed ~11h on one seat).

_send_path_degraded already tracks exactly this (set at polling
generation start, cleared on the first confirmed getUpdates round-trip),
so:
- BasePlatformAdapter._mark_connected() takes a `degraded` flag and
  publishes "retrying" instead of "connected" when set.
- TelegramAdapter passes its current _send_path_degraded into
  _mark_connected().
- The gateway/run.py reconnect watcher checks the same flag before
  overwriting the adapter's own status write.
- _record_polling_progress() republishes "connected" the moment
  polling actually proves progress, so the state does not stay wedged
  at "retrying" until the next disconnect.

Fixes #101391
2026-09-03 03:47:38 +05:30
Teknium ae70536485 Merge branch 'simp/web' into simp/integration 2026-09-02 15:01:45 -07:00
Teknium 01b523ee96 Merge branch 'simp/adapters' into simp/integration 2026-09-02 14:19:41 -07:00
Teknium a07dceb01f refactor(adapters/small_group): 5147->3565; line/email/ntfy/homeassistant/sms send-family and standalone-send dedupe, god-method extraction, dead find_pending_for_chat removed 2026-09-02 14:06:48 -07:00
Teknium c4f96565d0 refactor(adapters/whatsapp_webhook): 6449->4465; whatsapp_cloud/common/plugin send + media unification, webhook route/filter dispatch tables 2026-09-02 14:06:38 -07:00
Teknium 791417590d refactor(adapters/matrix_group): 9272->6311; matrix dead fetch_history/_send_simple_message/HTMLParser stubs removed, simplex/mattermost/irc one-verb API helpers and send builders 2026-09-02 14:06:37 -07:00
Teknium 8f1ef66b1f refactor(adapters/p2p_group): 12752->8794; buzz/photon/a2a/raft dedupe (sidecar paths unified in photon package, JSON-RPC helpers, Nostr event builders), dead a2a security getters + stream_message removed 2026-09-02 14:06:36 -07:00
Teknium b8bea7199c refactor(adapters/cn_group): 12368->8549; wecom split into streaming/media/send_queue mixins, dingtalk inbound parsers -> inbound.py, google_chat cards.py + setup_files.py, teams summary_writer.py; dedupe token/URL/credential helpers 2026-09-02 14:06:34 -07:00
Teknium 513ba5b846 refactor(adapters/feishu): 8095->5665; unify request/response wrappers, card builders, comment timeline selectors; dead symbols dropped; compact docs 2026-09-02 14:06:33 -07:00
Teknium 33c74a26c6 refactor(adapters/slack): 10667->7456; drop F811 _ssrf_redirect_guard shadow, split _handle_slack_message_impl (1135->~500) and connect(), unify Block Kit handlers, client/DM lookups, ts-eviction, upload retry; compact docs 2026-09-02 14:06:33 -07:00
Teknium 6e5b084b8b refactor(adapters/discord): 11250->7733; _HermesView base for 6 component views (1074-line factory -> 634), _send_prompt/_resolve_channel/_send_url_media/_send_local_file unification, opus loading and setup helpers extracted, dead _discord_allow_any_attachment removed 2026-09-02 14:06:32 -07:00
Teknium cc11c6bfdd refactor(adapters/telegram): 11957->8819; unify send/edit retry classification, media caching, auth gates, model-picker branches, polling recovery; compact docs 2026-09-02 14:06:32 -07:00
kshitijk4poor 9954445fea fix(feishu): create the dedup persist lock lazily for bare-constructed adapters
tests/gateway/test_feishu_bot_admission.py builds FeishuAdapter via
object.__new__ and installs dedup state by hand (install_dedup_state), so
the asyncio.Lock added in __init__ was missing -> AttributeError in CI.
Resolve the lock through a getattr-with-create helper and add it to the
shared test helper.
2026-09-03 02:17:01 +05:30
kshitijk4poor 1730a2c32a fix(gateway): serialize offloaded persists so concurrent flushes land in order
Offloading the fsync to a worker thread (previous commit) lets two inbound
messages have two persists in flight. Without ordering, a slow first flush
lands after the second and the disk file loses the newer token/id/seen-id
(in-memory state stayed correct, so the loss only surfaced after restart).
The loop-side dict could also be mutated while the worker iterated it.

- weixin ContextTokenStore / discord non-conversational tracker: snapshot
  the payload on the loop, hand it to the worker, serialize with a per-store
  asyncio.Lock.
- feishu dedup: same asyncio.Lock around the offloaded persist (its snapshot
  was already taken under _dedup_lock; the write was not).
- Regression tests: two concurrent persists with a slow first write must
  leave the union on disk. All three fail on the previous commit.
2026-09-03 02:17:01 +05:30
pierrenode cf6547da47 fix(gateway): offload remaining atomic_json_write calls on platform-adapter hot paths
atomic_json_write() calls os.fsync(), which blocks until the write
reaches stable storage. #83906 offloaded this in channel_directory.py,
gateway/run.py, and gateway/slash_commands.py, but the same blocking
pattern was still present on three platform-adapter message hot paths
that were never touched by that pass:

- plugins/platforms/feishu/adapter.py: _is_duplicate() persists the
  dedup cache on every new inbound message (shared by both websocket
  and webhook transports). The in-memory bookkeeping stays under
  _dedup_lock; the fsync now runs off it via asyncio.to_thread so
  concurrent callers on the ws-thread loop aren't blocked on disk I/O.

- gateway/platforms/weixin.py: ContextTokenStore.set() persists the
  context-token cache on every inbound message carrying a
  context_token, called from the async poll-loop handler
  _process_message.

- plugins/platforms/discord/adapter.py: mark_many() persists the
  non-conversational message-id tracker, called from both send() and
  send_update_prompt() when a message is tagged nonconversational.

Each fix mirrors the existing asyncio.to_thread(atomic_json_write, ...)
pattern from #83906. Regression tests assert the persist call runs on
a different thread than the event loop, mirroring
test_directory_write_runs_off_event_loop_thread in
test_channel_directory.py.
2026-09-03 02:17:01 +05:30
kshitijk4poor e0567770f3 fix(gateway): off-loop review follow-ups for #101603 / #101605
Findings from the efficiency review pass on the two salvages, applied as one
small follow-up:

- copilot_auth: check the negative cache BEFORE taking the per-fingerprint
  exchange lock. During the 60 s post-failure window, dashboard polls now
  raise immediately instead of parking an executor thread behind the
  in-flight holder (up to ~50 s) to learn the same answer. Test hangs
  without the check (timeout 124), passes with it.
- buzz _localize_inbound_media: download_path.read_bytes() was still
  evaluated on the loop as the argument to the offloaded cache call — up to
  the 128 MiB inbound cap. Read off the loop too.
- test_list_credential_pool_keeps_loop_responsive: 0.5 s block / 0.25 s
  threshold (2x margin) so runner descheduling cannot false-fail it while a
  real regression still trips it.
2026-09-03 02:07:05 +05:30
Teknium 0af4d3682f refactor(web): kanban plugin_api — unify status dispatch, run/board helpers, _errors_to_500 2026-09-02 13:33:24 -07:00
Teknium 800648ddc0 refactor(web): achievements plugin_api — table-driven catalog builder, shared json/session helpers 2026-09-02 13:33:24 -07:00
Teknium 313b244e8f refactor(plugins/model-providers): reuse agent.reasoning_effort clamps, per-model dict tables, compact profiles 2026-09-02 13:30:28 -07:00
Teknium 5d78bd817d refactor(plugins): disk-cleanup and security-guidance — table-driven scans/patterns, safety lists unchanged 2026-09-02 13:30:10 -07:00
Teknium 517f593af9 refactor(plugins/google_meet): run_bot extraction, subcommand tables, shared json-file helper, node protocol dedupe 2026-09-02 13:30:10 -07:00
Teknium 74edfe04e9 refactor(plugins/spotify): action dispatch table, single request helper 2026-09-02 13:30:10 -07:00
Teknium eb60dd519e refactor(plugins/teams_pipeline): subcommand dispatch, unified download/Graph helpers, compact models 2026-09-02 13:30:10 -07:00
Teknium 238260e6aa refactor(plugins/dashboard_auth): shared session/JWT/config helpers in _shared; compact basic/drain/nous/self_hosted 2026-09-02 13:30:10 -07:00
Teknium 191cc8d30a refactor(plugins/web): BaseWebSearchProvider + _common helpers across all search providers; keyless_mcp failover cleanup 2026-09-02 13:30:10 -07:00
Teknium e43f381b4e refactor(plugins/browser): shared BaseCloudBrowserProvider for browser_use/browserbase/firecrawl 2026-09-02 13:30:10 -07:00
Teknium 75b7e054a5 refactor(plugins/video_gen): fal/xai — shared model tables, submit/poll/download helpers, dedupe validation 2026-09-02 13:30:10 -07:00
Teknium b5b62aa83d refactor(plugins/image_gen): shared _common (config/key/save/error/picker helpers), convert all 8 providers, per-model tables 2026-09-02 13:30:10 -07:00
Teknium 994e844388 refactor(plugins/observability): langfuse — extract capture/redaction and trace-state lifecycle modules with deduped observation helpers 2026-09-02 13:30:10 -07:00
Teknium 57c35858e3 refactor(plugins/memory): holographic/supermemory/retaindb/byterover — action dispatch tables, unified HTTP helpers, dead code removal; provider registry dedupe 2026-09-02 13:30:10 -07:00
Teknium 87a450145c refactor(plugins/memory): mem0 — table-driven setup prompts, tool dispatch, compact backend 2026-09-02 13:30:10 -07:00
Teknium 64f430a051 refactor(plugins/memory): honcho — extract session context/auth/peers/migration, dialectic, client cache, tool schemas; unify _parse_* config helpers; table-driven CLI 2026-09-02 13:30:09 -07:00
Teknium 3cd4d6d69a refactor(plugins/memory): hindsight — split settings/embedded/setup modules, unify config parsing, remove dead helpers 2026-09-02 13:30:09 -07:00
Teknium 365485a9f6 refactor(plugins/memory): openviking — remove dead search/rebuild/migrate paths, extract setup/CLI with dedupe, action dispatch tables 2026-09-02 13:30:09 -07:00
Teknium b54cc716c7 refactor(plugins): shared path-based plugin loader; dedupe cron_providers/context_engine discovery; compact plugin_utils/plugin_storage 2026-09-02 13:30:09 -07:00
Teknium f0f7f55f76 refactor(web): dashboard_auth/web_routers/kanban dedupe + _common helpers (resume — verified partial work) 2026-09-02 13:30:05 -07:00
kshitijk4poor 568b16122d fix(gateway): finish the media-cache offload sweep (teams NameError, buzz, photon)
The previous commit removed cache_media_bytes from teams/adapter.py's import
block but left one bare call at the Bot-Framework image path — a NameError
that the surrounding 'except Exception' swallowed, so every BF image
attachment was silently dropped (ruff F821). Route it through
cache_media_bytes_async like the file's two other sites.

Same bug class at the sites the sweep did not reach:
- buzz _download_attachment / _localize_inbound_media: cache_media_bytes
  called from async methods → cache_media_bytes_async.
- photon _dispatch_inbound: _normalize_binary_payload (base64 decode +
  cache write of possibly multi-MB payloads) ran on the loop → closure made
  async, helper offloaded via asyncio.to_thread.

Tests: cover cache_media_bytes_async (thread-id + kwarg forwarding);
teams/buzz tests re-pointed at the async seam.
2026-09-03 01:58:56 +05:30
fangliquanflq cb0b66c161 perf(gateway): offload inbound media cache writes 2026-09-03 01:58:56 +05:30
Alexander Prendota 1bd0b8ed41 feat(providers): let an external-process provider ship out of tree
An external-process provider is an agent CLI Hermes drives over stdio rather
than an HTTP endpoint. Three things about it were spelled out for one vendor,
and each was a hard stop for any other:

* ``resolve_provider()`` gates on ``PROVIDER_REGISTRY``. Its auto-extend from
  ``providers/`` covered api-key providers only, so an external-process profile
  never entered it and ``hermes -m <that provider>`` died with "Unknown
  provider" before a client was ever built.
* ``resolve_runtime_provider()`` keyed the external-process branch on the
  literal ``"copilot-acp"``, so anything else silently fell through to the
  OpenRouter default instead of its own runtime.
* ``resolve_external_process_provider_credentials()`` hardcoded the binary
  (``copilot``), the argv (``--acp --stdio``), the env var names and the
  placeholder api_key — so a third-party provider would have been handed
  another vendor's CLI.

Now the profile carries what only the provider knows — ``process_command``,
``process_args``, ``process_command_env_vars``, ``process_args_env_var`` — and
the three core paths key on ``auth_type == "external_process"`` instead of a
name. copilot-acp's values move into its profile verbatim, so
``HERMES_COPILOT_ACP_COMMAND`` / ``COPILOT_CLI_PATH`` /
``HERMES_COPILOT_ACP_ARGS`` and its ``copilot-acp`` api_key placeholder behave
exactly as before; the new tests assert that alongside the out-of-tree case at
every step.

The error for a missing binary now names the provider and its own env override
instead of telling every user to install GitHub Copilot CLI.

Co-Authored-By: Junie <junie@jetbrains.com>
2026-09-02 09:57:39 -07:00
Alexander Prendota 1131b22856 feat(providers): let a provider profile supply its own client
``create_openai_client`` was a hardcoded if-ladder: copilot-acp builds an ACP
stdio shim, gemini builds a native client, everything else gets an
``openai.OpenAI``. There was no extension point, so a provider whose wire
protocol is not OpenAI-over-HTTP could only be added by editing this function —
which is exactly why an ACP provider cannot ship outside this tree today, even
though ``providers/__init__.py`` has discovered out-of-tree profiles from
``~/.hermes/plugins/model-providers/`` and pip entry points for a while.

``ProviderProfile.create_client(**client_kwargs)`` closes that gap. It returns
``None`` by default, so every provider that wants the standard client is
unaffected and the existing ladder still runs as the fallback. copilot-acp is
migrated onto it — its hardcoded branch is gone and its profile supplies the
client in three lines, which is the same three lines an external package writes.

Resolution goes by provider name first, then by ``base_url`` prefix, so a
runtime configured only by URL still reaches its profile — matching what the
replaced ``startswith("acp://copilot")`` branch did. A profile that raises is
logged and skipped: a third-party plugin can fail to provide a client, but it
cannot take the turn down.

Also replaces the two ``isinstance`` checks in ``agent/auxiliary_client.py``
that mean "this client is complete, do not wrap it" with capability flags the
client class declares — ``HERMES_SKIP_TRANSPORT_WRAP`` and
``HERMES_SKIP_ASYNC_WRAP``, mirroring ``SUPPORTS_HERMES_TOOL_CALLS`` in
``background_review.py``. Two in-tree consumers (the ACP shim and the Gemini
native client), an out-of-tree client is covered by the same declaration, and
the hot path no longer imports those modules just to type-test.

Co-Authored-By: Junie <junie@jetbrains.com>
2026-09-02 09:57:39 -07:00
Teknium 0ee98eda52 feat(models): add google/gemini-3.8-flash to nous + openrouter catalogs
Slots above gemini-3.7-flash (kept) in OPENROUTER_MODELS and
_PROVIDER_MODELS["nous"]; openrouter plugin fallback_models bumped
3.7 -> 3.8; model-catalog.json regenerated.

Verified live with test completions on both Nous Portal and OpenRouter
(model echo + billed). Same 1,048,576 window / 65,536 output / pricing
as 3.7-flash, so provider-agnostic metadata resolves via the existing
gemini entries and both routes bill live (official_models_api) — no
pricing snapshot needed.

Scoped to the two named providers: vertex/gemini/kilocode/gmi curated
lists, setup.py samples, and aux defaults untouched.
2026-09-02 09:46:15 -07:00
Teknium 001b8abbd4 fix(matrix): pin the E2EE crypto store per profile at connect(), not import
The multiplex gateway imports plugins/platforms/matrix/adapter.py once, so
the module-level _STORE_DIR/_CRYPTO_DB_PATH resolved against the root
HERMES_HOME for every profile: all bots' Olm identities landed in one
crypto.db and inbound E2EE failed with "no session found" (#89168).

connect() runs inside _profile_runtime_scope, so resolve the store dir
there via get_hermes_dir (honors the context-local HERMES_HOME) and cache
it on the instance -- diagnostics and error-log paths read outside the
scope then still report the store actually in use. Mirrors the
pairing-store fix (a6397c379).

Salvage of #89169 (per-call resolvers collapsed into one cached resolve;
dead `_CRYPTO_DB_PATH = None` alias dropped -- no external importers).
Also routes the last raw MATRIX_HOMESERVER read in check_matrix_requirements
through _startup_env_secret like its token/password neighbours (#69943).

Fixes #89168

Co-authored-by: Michael Short <18595461+mjshorty@users.noreply.github.com>
2026-09-02 07:01:23 -07:00
Teknium 9be1168cd3 fix(wecom): scope WECOM_WEBSOCKET_URL like its neighbours
Same class as #100627's WECOM_BOT_ID: the one remaining raw os.getenv in
WeComAdapter.__init__ let a secondary multiplex profile pick up the
default profile's bridged websocket URL. Route it through
_get_scoped_secret; folded into the existing scoped-miss test.
2026-09-02 07:01:23 -07:00
Teknium 2bcbdb61a7 fix(simplex): scope SIMPLEX_* reads to the active profile under multiplexing
SimplexAdapter.__init__ (auto_accept, group_allowed), the registry gates
check_requirements/validate_config/is_connected, _env_enablement and
_standalone_send all read SIMPLEX_* via raw os.getenv. Under
gateway.multiplex_profiles those paths run inside a secondary profile's
scope where os.environ holds the DEFAULT profile's YAML-to-env bridge
output -- so a secondary profile that never configured SimpleX was
auto-enabled on the default's daemon URL and inherited its group
allowlist / auto-accept setting.

Route every read through the module-local `_get_scoped_secret` wrapper
(get_secret; UnscopedSecretError -> os.getenv for the default profile,
which constructs unscoped) -- the same helper the IRC/ntfy/Photon/
Mattermost siblings use. Unlike the extra-only `_scoped_platform_setting`
shape proposed in #100241, this honors BOTH the secondary profile's own
.env (the scope) and its config.yaml extra, and needs no config.yaml
re-read in check_requirements.

Rewrite of #100241.

Co-authored-by: nftpoetrist <264138787+nftpoetrist@users.noreply.github.com>
2026-09-02 07:01:23 -07:00
nftpoetrist 07ee457a21 fix(wecom): scope WECOM_BOT_ID reads to the active profile under multiplexing
WeComAdapter.__init__ read WECOM_BOT_ID via a raw os.getenv() call, while
the immediately adjacent line for WECOM_SECRET already used the module's
_get_scoped_secret() helper. Under gateway.multiplex_profiles, a secondary
profile's adapter is constructed inside a scoped context where os.environ
still holds the DEFAULT profile's env-bridge output -- so a secondary
profile's bot would silently connect using the default profile's bot_id
while (correctly) using its own secret, or vice versa on a scope miss.

Switch the bot_id read to _get_scoped_secret(), matching the sibling
_secret/_dm_policy/_group_policy/allow_from reads in the same __init__
that were already migrated in #76664/#93545. _standalone_send's
out-of-process fallback branch constructs a fresh WeComAdapter(pconfig)
and therefore inherits this fix automatically -- no separate change
needed there.

Adds two regression tests to the existing TestWeComAdapterAuthzScope
class (already covering dm_policy/allow_from scoping per #93522),
mirroring its established fixture/assertion style. Mutation-verified:
both fail against the pre-fix code (asserting the default profile's
bot_id leaks into a secondary profile's scope) and pass with the fix.
2026-09-02 07:01:23 -07:00