- discord/telegram/slack/matrix standalone senders read bot tokens via
get_secret (in-turn: they run inside an installed scope — cron scheduler
and delegate spawns propagate it), never borrowing another profile's
env-bridged token under multiplex.
- telegram webhook-secret and matrix access-token/password startup reads
use the Slack pattern (#59739): get_secret, falling back to os.environ
only on UnscopedSecretError.
Follow-up to the salvaged #59076 commit:
- Replace the bare get_secret import with a module-level Slack-pattern
helper (_get_esecret): try get_secret, on UnscopedSecretError fall back
to os.getenv. The DEFAULT profile's email adapter constructs UNSCOPED
under multiplexing, where a bare get_secret raises and would crash the
email path on startup — the exact WhatsApp defect fixed in 5438e9c629
(whatsapp_common._get_wsecret).
- Extend scope coverage to the remaining scope-blind reads:
EMAIL_IMAP_PORT / EMAIL_SMTP_PORT / EMAIL_POLL_INTERVAL (_esecret_int
replacing utils.env_int) and EMAIL_TRUST_FROM_HEADER (_esecret_bool
replacing utils.env_bool).
- Add tests: default-profile unscoped-under-multiplex construction, and
scoped ports/trust-flag no-environ-inheritance.
The email adapter (plugins/platforms/email/adapter.py) read
EMAIL_ADDRESS, EMAIL_PASSWORD, EMAIL_IMAP_HOST, EMAIL_SMTP_HOST,
EMAIL_ALLOWED_USERS, and EMAIL_ALLOW_ALL_USERS via os.getenv()
directly. In a multiplexed gateway, os.environ holds the default
profile's .env values, so every secondary profile inherited the
default profile's email credentials instead of its own.
This was a sibling of the api_server env-leak bug (#52307/#50051):
the same os.getenv→get_secret migration that PR #50094 applies to
gateway/config.py, but for the email adapter itself, which neither
PR #50094 nor #51374 covers.
Changes:
- plugins/platforms/email/adapter.py: replace os.getenv with
agent.secret_scope.get_secret for all EMAIL_* credential reads
(adapter __init__, check_email_requirements, _allowlist_in_effect,
_dispatch_message allowlist gate, _send_email SMTP helper).
- gateway/config.py: add _getenv/_getenv_str/_getenv_int helpers
(from PR #50094) and replace os.getenv with _getenv for the email
block in _apply_env_overrides, so config.platforms[EMAIL].extra
is populated from the scoped value.
- tests/gateway/test_email_secret_scope.py: 5 new tests covering
scoped credential reads, environ fallback without scope, missing-
key-no-leak, allowlist scoping, and check_email_requirements scoping.
Related: #50051, #52307, PR #50094, PR #51374
FEISHU_APP_SECRET/FEISHU_ENCRYPT_KEY/FEISHU_VERIFICATION_TOKEN, WECOM_SECRET,
PHOTON_PROJECT_SECRET/PHOTON_SIDECAR_TOKEN (adapter + auth.load_project_credentials)
and BUZZ_PRIVATE_KEY now read through _get_scoped_secret.
NTFY_TOKEN, HASS_TOKEN, TWILIO_ACCOUNT_SID/TWILIO_AUTH_TOKEN and
DINGTALK_CLIENT_SECRET now read through _get_scoped_secret. Also replaces
the SMS adapter's bare os.environ["TWILIO_AUTH_TOKEN"]/["TWILIO_ACCOUNT_SID"]
__init__ reads (KeyError-prone) with helper reads defaulting to "".
Route IRC_SERVER_PASSWORD/IRC_NICKSERV_PASSWORD, LINE_CHANNEL_ACCESS_TOKEN/
LINE_CHANNEL_SECRET, TEAMS_GRAPH_ACCESS_TOKEN/TEAMS_CLIENT_SECRET and
MATTERMOST_TOKEN reads at __init__/availability/standalone-send time through
a module-level _get_scoped_secret helper (get_secret, UnscopedSecretError ->
os.getenv fallback), mirroring whatsapp_common._get_wsecret / Slack #59739.
Scoped miss returns the default — no cross-profile environ borrow.
Review follow-ups on the #76357 salvage:
- _message_reference_from_ids + _reply_reference_for_send collapse the
3x duplicated MessageReference construction (naming mirrors telegram's
_reply_to_message_id_for_send).
- The overflow elif's comment claimed PartialMessage has no to_reference;
discord.py 2.7.1's PartialMessage does (message.py L1901) — the branch
is belt-and-suspenders for duck-typed priors, now labeled as such.
Every reply paid one extra Discord API round trip: the text send path,
the voice send path, and the edit path each called fetch_message() just
to obtain a reference or an editable handle. Discord resolves
message_reference payloads from ids alone, and PartialMessage.edit()
works without a fetch — so build MessageReference directly (with
fail_if_not_exists=False, preserving the deleted-target behavior the
existing send-side 10008 retry already covered) and use
channel.get_partial_message() for edits. Overflow continuations keep
threading via an ids-built reference fallback for PartialMessage.
Measured by call count (deterministic): reply sends and edits now make
ZERO fetch_message calls where they made 1 per reply and 1 per edit
(including every streaming edit tick).
Tests: pin that first-mode replies construct the reference without any
fetch, deleted-target retry test updated to assert fetch await_count==0
(retry now happens purely send-side), overflow/edit mocks retargeted
from fetch_message to get_partial_message (any fetch regression breaks
all five). Note: 4 discord-suite failures are pre-existing ordering
flakes — identical with the change stashed on clean main.
Follow-ups on the #75382 salvage (review findings):
- _wenv/_get_wsecret now catch UnscopedSecretError and fall back to
os.getenv for the DEFAULT profile's adapter, which constructs and sends
outside any _profile_runtime_scope under multiplexing — a bare
get_secret would crash its WhatsApp path (fixing one profile by
breaking another). Same pattern as Slack SLACK_APP_TOKEN (#59739) and
the Matrix recovery key. Scoped misses still return the default — no
cross-profile borrow.
- bridge_env overlay extended to the full WHATSAPP_* set bridge.js
consumes (DEBUG, FORWARD_OWNER_MESSAGES, REPLY_PREFIX,
MAX_MESSAGE_LENGTH, CHUNK_DELAY_MS, SEND_TIMEOUT_MS).
- Removed the always-true conditional on WHATSAPP_MODE injection.
Fix#75349
Root cause:
Under multiplex_profiles, secondary profiles run inside
_profile_runtime_scope which installs a per-profile secret scope via
set_secret_scope. The WhatsApp adapter (and the shared
WhatsAppBehaviorMixin + Cloud API adapter) read WHATSAPP_MODE,
WHATSAPP_DM_POLICY, etc. via raw os.getenv(), bypassing the secret
scope. Since os.environ doesn't contain secondary profile .env values,
the bridge silently falls back to 'self-chat' and rejects all inbound
messages with self_chat_mode_rejects_non_self.
Fix:
- Add _wenv() helper in adapter.py that reads WHATSAPP_* vars through
get_secret() (agent.secret_scope), which honors the active scope.
- Replace all os.getenv('WHATSAPP_*') calls in adapter.py,
whatsapp_common.py, and whatsapp_cloud.py with get_secret()-based
equivalents.
- Inject resolved WHATSAPP_* values into the bridge subprocess
environment so the Node.js bridge (which reads process.env) sees the
profile's own configuration.
Changes:
- plugins/platforms/whatsapp/adapter.py: 37 lines (+ helper, bridge_env
injection, 2 os.getenv→_wenv)
- gateway/platforms/whatsapp_common.py: 13 lines (6 os.getenv→_get_wsecret)
- gateway/platforms/whatsapp_cloud.py: 21 lines (9 os.getenv→_get_wsecret)
- New regression test: 6 test cases covering scope isolation, fallback,
and cross-profile non-leakage.
The Matrix adapter read MATRIX_RECOVERY_KEY via os.getenv, so under
gateway.multiplex_profiles every profile resolved the default profile's
key. That produced "recovery key verification failed: Key MAC does not
match" and broke E2EE for secondary profiles (#69090).
Route the read through agent.secret_scope.get_secret, which honors the
active profile's scope, with an os.getenv fallback for an unscoped read
under multiplex (default-profile startup loop) — mirroring the Slack
app-token pattern (#59739). Applied to both the startup verification
site and the status diagnostic.
Fixes#69090
Track which source seeded the DM allowlist so live intake does not let a
stale env carrier override explicit config, while env-seeded adapters still
reread pairing mutations.
Teknium review on #62947: os.environ.clear()/update around deferred
loaders is unsafe under concurrency and misses teams_pipeline's direct
adapter import.
Defer microsoft_teams binding in the Teams adapter, no-op
dotenv.load_dotenv while the SDK imports, keep api_server explicit
disable, and add SDK-import + load_gateway_config canaries.
Fixes#62935
Under gateway.multiplex_profiles, Discord and Telegram authorization gates
(allowed/ignored channels, allowed users/roles, allow-all flags) were read
from process-global os.environ, populated first-writer-wins by the YAML->env
bridge in each adapter's _apply_yaml_config. The first profile to initialize
pinned its allow/deny lists — and its ALLOW_ALL flags — for every other
profile in the process (issue #72348, incl. the Telegram mirror reported in
the thread).
Fix (per-adapter-instance gate reads, whole class):
- gateway/authz_mixin.py: new _platform_gate_env — scope-authoritative gate
read: under an installed profile secret scope with multiplex active, a
missing key returns the default instead of falling through to os.environ
(which may hold another profile's value). Single-profile behavior is
byte-identical to os.getenv.
- Discord adapter:
- connect() snapshots all gate env vars (_GATE_ENV_KEYS) inside the owning
profile's runtime scope into a per-adapter dict; new accessors
(_get_allowed_channels/_get_ignored_channels/_get_allowed_users/
_get_allowed_roles/_get_no_thread_channels/_discord_allow_all_users/
_gateway_allow_all_users/_get_allow_bots) resolve snapshot -> config.extra
-> scope-aware env, replacing every raw os.getenv gate read: on_message
channel gates, _is_allowed_user allow-all flags, slash authorization,
fail-closed diagnostics, missed-message backfill, bot-message gating,
and _component_check_auth (component buttons).
- _apply_yaml_config always seeds gate values into PlatformConfig.extra
(incl. new allowed_roles / allow_all_users keys) and SKIPS the
process-global env writes when loading a profile-scoped config under
multiplex; the legacy first-writer env bridge is preserved verbatim for
single-profile deployments.
- _resolve_allowed_usernames no longer unconditionally rewrites
os.environ[DISCORD_ALLOWED_USERS] — under multiplex the resolved IDs stay
adapter-local (snapshot refresh); single-profile keeps the env rewrite.
- Telegram adapter (mirror of the same class): intake prefilter and
callback-auth fallbacks, _telegram_auth_env_configured, and the
allowed/ignored chats-topics-threads getters now read via the scoped gate
reader; _apply_yaml_config skips authorization env writes for
profile-scoped loads and seeds free_response_chats/ignored_threads extras.
Regression tests (tests/plugins/platforms/test_discord_gate_isolation.py):
two adapter instances with different allow-lists enforce their OWN lists
order-independently across message, slash, and component gates; negative
allow-all case proves profile A's open-access flag cannot authorize profile
B; username-resolution env-clobber; YAML-bridge seeding/skip matrix; and the
Telegram scoped-reader matrix. Sabotage-verified: reverting either the
Discord snapshot accessors or the Telegram scoped reader fails 12/2 tests
respectively.
Credit: builds on the per-adapter accessor direction of PR #72427
(@JonthanaHanh) and the scope-aware-reader approach validated live on v0.19.0
by @yournetworkplug-ctrl for the Telegram mirror; scope corrections from
jackjin1997's and cal88's analysis in the issue thread (allow-all flags,
unguarded username-resolution env write, per-site channel reads).
Fixes#72348
The #40863 intake prefilter rejected unauthorized DMs whenever an allowlist
existed, so gateway pairing never ran even when the operator set
telegram.unauthorized_dm_behavior: pair (which must win over the #9337
allowlist silence default). Pass those DMs through; groups stay blocked.
Google API and authentication packages permit vulnerable httplib2 and pyasn1
transitives, while the Workspace and Google Chat runtime installers previously
treated any importable version as sufficient. Existing environments could
therefore remain vulnerable after the project dependency pins were repaired.
Carry the fixed versions through the Google and Vertex extras, lazy feature
requirements, lockfile, and both runtime installers. Route the documented
Google Chat installation path through its maintained secure requirements
instead of an unconstrained direct pip command.
Detect stale distributions, install only unsatisfied requirements, and verify
the result before continuing. Behavioral tests cover those repair invariants
without freezing manifests, lockfiles, or complete package sets.
Related #72108
Extracted from #72840
Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
524ab5399 widened the media read timeout from send_video to "all upload send
paths" - send_voice/send_audio/send_photo/send_document/send_media_group/
send_animation. Both send_photo calls inside send_image() were missed, so
they still ran on the short timeout the rest of the Bot API is tuned for
while the sibling media paths already pass it.
The missed pair is the worst one to miss: send_image tries a URL send first,
then falls back to downloading the image and uploading the bytes - the path
documented as "supports up to 10MB", i.e. the slowest send in the file and
the one whose server-side processing wait most often outlasts the short
budget. When it times out the handler's last resort posts the bare URL as
text, so the picture silently never arrives as a picture.
Pass _MEDIA_SEND_READ_TIMEOUT on both, covered by two behavioral tests that
drive send_image for real - the URL send and the forced byte-upload fallback
- and assert the read_timeout that actually reaches the Bot API.
Follow-up to PR #70238. Remove 'free-response channel' and
'authorization' jargon from the model-facing prompt. Collapse the
triple-negative 'do not ask / do not reject / do not stay silent'
into a single directive. ~70 tokens vs ~175 in the contributor's
version, same semantics.
send_video got the 60s read_timeout but send_voice/send_audio/send_photo/
send_document/send_media_group/send_animation upload through the same PTB
request path and hit the same server-side processing wait before the
response arrives. Same class, all sites: they all pass
_MEDIA_SEND_READ_TIMEOUT now. Also drops an unused test helper.
Use database.journal_mode as the sole non-secret operator setting, preserve the vulnerable-SQLite safety gate and existing WAL databases, validate explicit DELETE results, document the active config path, and cover real SQLite openers with behavioral tests.
Add HERMES_JOURNAL_MODE env / database.journal_mode config for
virtiofs/NFS/SMB where WAL is not crash-safe. Route 5 bypass openers
through apply_wal_with_fallback so a single setting covers every .db
(#68545).
Telegram clients split messages above 4096 chars into multiple updates. A
long '/queue <prompt>' paste arrives as a COMMAND chunk near the limit plus
plain TEXT continuation chunk(s). _handle_command dispatched the command
chunk immediately, so the continuation landed as a separate plain message
that interrupted the running agent instead of being queued.
Near-limit (>= _SPLIT_THRESHOLD) command chunks now route through the same
text-batching pipeline used for split plain-text messages, merging the
continuation before dispatch. Short commands (/stop, /approve, ...) keep the
immediate path and are never delayed.
The compile_mention_patterns promotion moved the 'patterns is None ->
return []' short-circuit into the shared helper, which meant the
telegram/dingtalk wrappers now evaluated self.name (via log_prefix=)
even on the no-patterns path. On main that path returned before
touching any adapter attributes; tests construct bare adapters via
object.__new__ that lack .platform, so TestTelegramGuestMentionGating
failed with AttributeError. Restore the early return in both wrappers
for exact behavior parity with main.
- base.on_processing_complete implements the opt-in remove-ack/add-outcome
flow driven by _OK_EMOJI/_FAIL_EMOJI class attrs and the
_add_reaction(chat_id, message_id, emoji)/_remove_reaction(chat_id,
message_id) primitive shape; default stays a no-op.
- photon drops its override (exact behavioral match).
- slack/discord/feishu/matrix/telegram/google_chat keep overrides: divergent
primitive signatures (team_id routing, raw message objects, reaction-id
handles, replace-semantics setMessageReaction) or extra state protocols.
The disease: ~15 scattered raw yaml.safe_load(config.yaml) reads that
silently miss managed-scope overlay, ${ENV_VAR} expansion, profile-aware
pathing, and root-model normalization. Every new config feature needed an
N-site sweep (incident chain 9cbcc0c9c8 → 732293cf87 → b0e47a98f9 →
1928aa0443). This commit assigns every raw read to an owner and adds a
lint-guard test so the class cannot regrow.
New primitive (additive-only change to hermes_cli/config.py):
read_user_config_raw(path=None) — reads the user file EXACTLY as
written; docstring states it is ONLY legal for write-back round-trips
and raw-file diagnostics. Behavioral reads must use
load_config()/load_config_readonly().
BEHAVIOR FIXES (class-a sites migrated to a canonical loader — these
previously read values that could DIFFER from the effective config):
gateway/run.py _try_resolve_fallback_provider → _load_gateway_runtime_config
keys: fallback_providers/fallback_model (provider, model, base_url,
api_key). Drift fixed: a managed-pinned fallback chain was ignored;
an api_key of "${OPENROUTER_API_KEY}" reached the resolver unexpanded.
gateway/run.py GatewayRunner._load_provider_routing → same loader
key: provider_routing. Drift fixed: managed-pinned routing prefs and
${VAR} templates were ignored.
gateway/run.py GatewayRunner._load_fallback_model → same loader
keys: fallback chain. Same drift as above.
gateway/run.py GatewayRunner._refresh_fallback_model
keeps the raw primitive (its last-known-good-on-parse-failure contract
forbids the fail-open loader, which returns {} on a torn write) but now
applies managed overlay + env expansion inline. Drift fixed: chain
edits under managed scope / env templates were previously frozen out.
tui_gateway/server.py _load_cfg (72 behavioral call sites)
now = raw read + managed overlay (pre-existing) + NEW ${VAR} expansion,
split from a new _load_cfg_raw() write-back primitive. Drift fixed:
e.g. custom_prompt: "hello ${VAR}", agent.system_prompt, model,
api_key/base_url templates reached sessions unexpanded. DEFAULT_CONFIG
is deliberately NOT merged (callers treat missing keys as unset;
`_load_cfg() == {}` sentinels and _save_cfg round-trips depend on it).
tui_gateway/server.py _profile_configured_cwd
keys: terminal.cwd of a NON-launch profile. Drift fixed: managed
overlay + ${VAR} expansion now apply (load_config() would resolve the
wrong profile's home, so the raw primitive + inline pipeline is used).
plugins/platforms/telegram/adapter.py _reload_dm_topics_from_config
→ load_config_readonly(). keys: platforms.telegram.extra.dm_topics.
Drift fixed: managed overlay + profile-aware pathing + expansion.
plugins/memory/holographic _load_plugin_config → load_config_readonly().
keys: plugins.hermes-memory-store.*. Same drift class.
WRITE-BACK ROUND-TRIPS (class-b: stay raw BY DESIGN via read_user_config_raw;
merging defaults/overlay would pollute the saved user file):
gateway/slash_commands.py: model persist x2, _save_gateway_config_key,
memory/skills write_approval toggles
gateway/platforms/yuanbao.py auto-sethome
tui_gateway/server.py _write_config_key + all cfg→_save_cfg blocks
(reasoning show/hide/full/clamp, details_mode[.section], prompt)
→ new _load_cfg_raw()
plugins/memory/holographic save_config
RAW-FILE DIAGNOSTICS + presence-sensitive bridges (class-c: stay raw,
now via the shared primitive with an explanatory comment):
hermes_cli/doctor.py x5 (model validation, stale-root-keys, .env drift,
deprecation sweep, memory-provider probe — the latter two keep their
inline managed overlay where they had one)
gateway/run.py _bridge_max_turns_from_config and the module-level
TERMINAL_*/HERMES_* env bridge (bridging merged defaults would export
all of DEFAULT_CONFIG into the environment; both keep their inline
overlay + expansion)
hermes_cli/send_cmd.py env bridge (same presence-sensitivity)
hermes_cli/gateway.py multiplex-conflict probe (reads the DEFAULT root's
config, not the active profile's — load_config is the wrong owner)
hermes_cli/profiles.py / hermes_cli/web_server.py / tools/wake_word.py
multi-profile reads (load_config targets only the ACTIVE profile home)
cron/jobs.py _resolve_default_model_snapshot and cron/scheduler.py
run_job config read keep their existing inline overlay+expansion but
now share the primitive (their fail-open + last-value semantics and
the deliberate no-defaults merge are preserved exactly).
Failure-semantics audit: every migrated site preserves its exact previous
behavior on missing file ({} / early return) and parse failure (raise into
the caller's existing except, warn, last-known-good, or fail-open) —
read_user_config_raw intentionally mirrors bare open()+safe_load semantics
(raises on parse errors, {} only on FileNotFoundError/non-dict root).
Guard: tests/hermes_cli/test_config_read_guard.py scans the tree for
yaml.safe_load within 6 lines of a 'config.yaml' reference outside an
explicit ALLOWLIST (hermes_cli/config.py, gateway/config.py, gateway/run.py
fallback path, hermes_cli/managed_scope.py which reads the MANAGED file,
gateway/readiness.py parse-health probe) and fails on new offenders.
E2E: tests/hermes_cli/test_config_loader_e2e.py runs a subprocess with a
temp HERMES_HOME (config.yaml containing ${E2E_PROMPT_SUFFIX}) plus a
HERMES_MANAGED_DIR overlay pinning agent.reasoning_effort, asserting
tui _load_cfg resolves "hello world"/"high" while _load_cfg_raw +
_save_cfg round-trip the template and user value verbatim with no
managed/default leakage.
LINE plugin.yaml plus line/wecom-callback/msgraph-webhook/
whatsapp-cloud/teams user-guide pages still documented the old
IPv4-only 0.0.0.0 defaults; update to the dual-stack unset default.
Telegram webhook env docs live in the adapter docstring (updated with
the code change); its plugin.yaml has no webhook host entry.
Same class of bug as the LINE adapter (NS-603): defaulting the webhook
bind to "0.0.0.0" (or hardcoding it) binds IPv4 ONLY, so the listener
is unreachable over IPv6-only private networks such as Fly.io 6PN.
- wecom callback_adapter: DEFAULT_HOST None; config.py env seed no
longer forces 0.0.0.0 when WECOM_CALLBACK_HOST is unset.
- msgraph_webhook: DEFAULT_HOST None; the allowed_source_cidrs
requirement still fires for the all-interfaces default (host=None is
treated as network-accessible).
- whatsapp_cloud: DEFAULT_WEBHOOK_HOST None.
- teams: hardcoded 0.0.0.0 TCPSite bind → _DEFAULT_HOST=None with new
TEAMS_HOST / extra.host override (mirrors LINE_HOST pattern).
- telegram: hardcoded listen="0.0.0.0" → default "" (tornado
bind_sockets opens one socket per address family; verified against
PTB 22.6/tornado) with new TELEGRAM_WEBHOOK_HOST / extra.webhook_host
override.
Explicit host overrides everywhere are preserved; empty/unset collapses
to the dual-stack default. "::" remains a bad substitute on
bindv6only=1 hosts (see LINE adapter comment).
The LINE adapter's webhook server defaulted to host="0.0.0.0", which
binds IPv4 ONLY. On IPv6-only private networks — notably Fly.io 6PN,
where the hosted edge router reverse-proxies LINE ingest to
<app>.internal:8646 over an fdaa: IPv6 address — nothing is listening
on the dialed address: connection refused → customer-visible 502 when
LINE's console verifies the webhook (NS-603).
This is the same bug the generic webhook adapter fixed in d542894ad;
the LINE adapter was never updated to match. Fix mirrors that commit:
- DEFAULT_HOST = None → asyncio/aiohttp create_server binds one socket
per address family (v4 + v6), regardless of the bindv6only sysctl.
"::" is NOT a valid substitute — Fly machines set bindv6only=1, so
it would yield an IPv6-only socket and break IPv4 loopback probes.
- Empty-string host collapses to None; LINE_HOST/extra.host still pin
a specific bind address.
- reuse_address=False scoped to macOS only (BSD wildcard-socket
traffic-splitting footgun), mirroring 9420ad946.
- The three outbound-media guards compared webhook_host == "0.0.0.0"
by string equality; extracted to _missing_public_url() which treats
None/0.0.0.0/::/"" as "no fetchable hostname" so the LINE_PUBLIC_URL
requirement still fires under the new default. _media_url() falls
back to 127.0.0.1 instead of interpolating 'None' into URLs.
Tests: dual-stack default decision table (default None, empty→None,
pinned preserved, LINE_HOST override), behavioural both-families bind
proof via runner.addresses, and the media public-URL guard matrix.
88 passed, ruff clean.
Companion router fix (hermes-agent-router) routes /webhooks/line and
/line/webhook|/line/media to :8646 over 6PN; both are needed for
end-to-end hosted LINE delivery.
Fixes NS-603
resolve_sidecar_dir() probes the filesystem (touch/unlink) and can mirror
sidecar files to HERMES_HOME. Doing that as a module-import side effect
meant plugin discovery, `hermes --help`, and test collection all paid a
filesystem probe (and possibly a mirror copy) just for importing the
photon adapter or CLI.
Convert _SIDECAR_DIR/_NPM_ERROR_LOG in adapter.py and cli.py to lazy
cached accessors (_sidecar_dir()/_npm_error_log()); resolution now
happens on first actual use. Existing tests that monkeypatch the
_SIDECAR_DIR module global keep working — the accessors honor a
non-None value. Adds a regression test proving import performs no
resolution.
The Photon iMessage sidecar needs node_modules under
plugins/platforms/photon/sidecar/, but hosted/managed images keep the
whole install tree under an immutable /opt/hermes — every install and
self-heal path (setup CLI, stale-deps reinstall, cold install) died on
EROFS, and hosted users have no shell to work around it.
Three-layer fix, mirroring the WhatsApp bridge resolver pattern:
1. Bake the deps into the image. The Dockerfile now runs npm ci for the
sidecar in the layer-cached dependency stage (deterministic installs
from the committed lockfile; the postinstall spectrum-ts patch runs
at build time). Hosted happy path needs no runtime install at all.
2. New sidecar_paths.resolve_sidecar_dir() decides where the sidecar
runs from: PHOTON_SIDECAR_DIR override > writable source dir (dev
installs, unchanged) > read-only dir with baked fresh deps (managed
image) > mirror to $HERMES_HOME/photon/sidecar (writable data
volume) when deps are missing or stale in a read-only tree. The
mirror refreshes changed source files on image updates while
keeping node_modules, so the existing lockfile-staleness self-heal
works there.
3. connect() can now cold-install: _start_sidecar() runs the bounded
npm ci bootstrap when node_modules is missing instead of raising
immediately, and check_requirements() reports available when a
self-install is possible (npm present + writable resolved dir) so
the gateway actually creates the adapter on hosted instances. A
failed bootstrap still raises the actionable error, which connect()
surfaces as the retryable SIDECAR_FAILED fatal state on the
dashboard.
Tests: resolver decision table (env override, in-place, mirror,
refresh, fail-open), cold-install lifecycle paths, and a Dockerfile
contract test guarding the baked-deps + no-chown invariants.
Fixes NS-606.
Maintainer rework of #45580 (issue #54036) on top of the contributor's
cherry-pick, which targeted spectrum-ts 3.1.0 while main pins 8.0.0:
Sidecar (primary detection, new):
- stream-staleness.mjs: pure decision rules, executable under node.
* classifyProbeRejection: only a not-found-shaped rejection of the
synthetic-id read counts as a completed round-trip (ALIVE); any other
rejection is INCONCLUSIVE — never alive. The original /probe treated
ANY rejection as alive, which was too loose.
* shouldProbe: probe only after 10+ min of stream silence (configurable
via PHOTON_STREAM_SILENCE_PROBE_MS; <=0 disables) with a cooldown.
* isZombieSuspect: zombie only on silence past threshold AND a
probe-proven live channel. Silence alone NEVER degrades (shared lines
can be quiet for hours); inconclusive probes NEVER degrade (network
may be down — the iterator will throw and the re-subscribe loop
recovers on its own).
- index.mjs: track last inbound-iterator yield (noteInboundYield), run a
30s watchdog tick, and on a confirmed zombie feed markStreamDegraded ->
the existing exit-75 restart path. /healthz gains a stream.staleness
block (silentForMs, threshold, lastProbeOutcome, zombieSuspected).
/probe reworked to strict semantics: 200 only on a proven round-trip,
503 with outcome hung|inconclusive otherwise.
Adapter (second layer, reworked):
- _probe_once returns tri-state alive|hung|inconclusive; only a hung
sidecar HTTP call counts toward the respawn counter — inconclusive
resets nothing and triggers nothing.
- default probe_interval_seconds 60 -> 600 (conservative; avoid restart
storms on quiet lines).
- _monitor_sidecar_health surfaces zombieSuspected from /healthz as a
warning; the fatal UPSTREAM_STREAM_DEGRADED path is unchanged and fires
when the sidecar escalates.
Tests: test_zombie_stream_watchdog.py executes the real node decision
module and drives the adapter against mocked /healthz responses;
test_presence_watchdog.py updated for the tri-state probe.
Also adds contributor mappings for nickkarhan (#53283) and vaibhavjnf
(#45580).
Follow-up to the URL markdown fix: extract the /send builder decision into
sidecar/send-format.mjs and rewrite test_url_send_path.py to execute the real
module under node (format+text in -> chosen builder out) instead of regex-
grepping index.mjs source, which is a banned test pattern in this repo.
spectrum-ts's live-stream consumer (consumeLive in spectrum-ts 3.x) only
reconnects when its inbound async iterator throws or ends. A half-open
("zombie") gRPC socket — where the TCP connection stays ESTABLISHED but the
peer is gone (NAT idle-timeout, network blip, laptop sleep) — makes the
iterator hang forever: no error, no end. The SDK exposes no gRPC keepalive
knob (createClient takes only {address, tls, token}; grpc.keepalive_time_ms
defaults to -1 = pings off), so the inbound stream silently dies and stays
dead until the gateway is restarted. Symptom: the agent's iMessage line goes
"online but deaf" — Photon's cloud-side fallback answers users with "the agent
isn't online right now" and inbound never reaches the gateway.
Fix, entirely in the code we own (no SDK fork):
- Sidecar gains a POST /probe endpoint that drives a cheap unary read
(space.getMessage on a synthetic id) over the SAME gRPC channel the inbound
stream uses. A live channel round-trips in ms (server returns not-found,
which is success for liveness); a zombie hangs. It sends nothing to any user
and creates no chat (space.get is local in shared/dedicated mode; only the
message read touches the wire).
- The adapter runs a presence watchdog: it probes on an interval, skips the
probe when natural inbound traffic already proved liveness within the
window, and after N consecutive failed probes respawns the sidecar — a fresh
Spectrum() re-subscribes the stream and re-registers presence. Successful
probes double as application-level keepalive, helping prevent the zombie
from forming at all. Respawn is lock-guarded against double-spawn and the
watchdog is torn down cleanly on disconnect.
Behavioural settings live in config.yaml (extra), bridged to env per the
.env-is-secrets-only convention:
probe_interval_seconds (60), probe_timeout_seconds (10),
probe_max_failures (3). A non-positive interval disables the watchdog.
Tests: tests/plugins/platforms/photon/test_presence_watchdog.py covers config
resolution, the disable switch, probe alive/dead(500)/timeout/no-client, the
core N-failures->one-respawn detection, success-resets-failures, stop-then-
start respawn ordering, and lock-guarding — all without spawning Node or
hitting the network.
Contributed by Vaibhav Sharma (X: @vabbyshabby).
imessage.effect.message crashed the sidecar at import against SDK stubs/
builds lacking the effect surface (caught by the patch-failure health test).
Optional-chain with {} fallback; /send-effect rejects cleanly instead.
Follow-up to the #48194 pick: it was written before #43665 landed and
re-added its own /send-poll sidecar route and poll import. Collapse the
duplicates:
- keep #43665's /send-poll route (>=2 trimmed string options) as the
single sidecar implementation; drop #48194's variant
- drop the duplicated poll import in the sidecar destructure
- make adapter.send_poll() a thin wrapper over _sidecar_send_poll(), the
one /send-poll client (shared with the poll-backed clarify path), and
align its validation to the sidecar's >=2-options contract
The `clarify` tool's multiple-choice prompts flattened to a numbered text
list on Photon/iMessage, even though iMessage has a native poll bubble and
spectrum-ts already exposes it via the `poll()` content builder. Two gaps
caused the flattening:
* Outbound: the sidecar only had `/send` (text); there was no way to send
a poll, so the base adapter's numbered-text fallback was used.
* Inbound: `normalizeContent()` handled only text/attachment/voice, so a
poll vote (`poll_option`) was dropped on the floor ("[Photon content
type not handled: poll_option]") and never resolved the clarify.
Fix, end to end:
* Sidecar: import `poll` from spectrum-ts; add a `/send-poll` route
(`space.send(poll(title, ...options))`); serialize inbound `poll_option`
(the vote: chosen title + selected bool) and `poll` content in
`normalizeContent()`.
* Adapter: override `send_clarify` — for choices, send a native poll via
`_sidecar_send_poll` and call `mark_awaiting_text` so the gateway's
existing pending-clarify text-intercept resolves the answer; open-ended
clarifies keep the plain-text path. Inbound `poll_option` selections are
dispatched as a plain-text MessageEvent carrying the chosen option
(deselections / empty votes are dropped). If the poll send fails (an
older sidecar without `/send-poll`, or a send error) it falls back to the
numbered-text clarify, so nothing regresses on a half-upgraded restart.
No new model tool, no new env var, no core change — the capability lives at
the platform edge. The poll vote reuses the existing clarify text-intercept
resolution path, so no new gateway resolution mechanism is introduced.
Tests: tests/plugins/platforms/photon/test_poll_clarify.py — inbound vote ->
choice text, deselection/empty-vote dropped, send_clarify sends a poll +
enables text-capture, open-ended stays text, and poll-failure falls back to
the text list. Full photon suite green.
Contributed by Vaibhav Sharma (X: @vabbyshabby).
Teknium