Files
hermes-agent/tests/gateway/relay/test_relay_media.py
T
Ben Barclay 29033a3fd5 fix(relay): restore voice-note STT — wire media[] MIMEs, message_type "voice", and a User-Agent for CDN downloads (#95274)
* fix(relay): map wire media[] → event.media_types; accept message_type voice

A relayed voice note arrived as MessageType.AUDIO with media_types=[] —
the STT gate (_event_media_is_stt_input) excludes AUDIO unconditionally
and its per-attachment MIME rescue was unreachable, so STT never fired
and the agent fell back to the "user sent an audio file attachment"
context note (live-verified on staging 2026-08-26, Discord + Telegram).

Two wire-boundary fixes, both additive within contract_version 1:

- "voice" parses to MessageType.VOICE: the enum already had it — pinned
  by test so a future refactor can't collapse the two.
- media[] is now mapped into event.media_types (positional alignment
  with media_urls; mime-less entries keep their slot as ""). This is
  what run.py's per-attachment classifiers key off, so EVERY relayed
  attachment — image vs document, audio vs voice — now routes like its
  native-adapter equivalent, not just voice notes.

Behaviour pinned: new-connector voice → STT-eligible; legacy
audio-typed events unchanged (no STT); music uploads never STT-eligible
(direct _event_media_is_stt_input assertions on real wire-parsed
events, not mocks).

Pairs with the gateway-gateway PR that puts "voice" on the wire.

* review: pin the STT gate by test; fail safe on media/media_urls mismatch

Addresses independent review of #95274.

1. The PR's acceptance criterion is STT ROUTING, but no committed test
   called _event_media_is_stt_input — it was only asserted ad-hoc. Adds
   TestSttGate: voice→eligible, voice-without-media_types→eligible
   (the new-connector/old-gateway shape), legacy audio-typed voice
   note→not eligible, music→not eligible. Mutation-verified: removing
   the VOICE branch from the gate turns these RED.

2. media_urls and media[] are INDEPENDENT wire fields that consumers
   index by the same i. Mapping MIMEs positionally without checking
   agreement means a disagreeing producer misassociates a MIME with the
   wrong URL and mis-routes that attachment — strictly worse than no
   MIME, which degrades safely to message-level classification.
   _media_types_from_wire() now maps only when the lengths agree, warns
   and returns [] otherwise.

Note for the record: MessageType.VOICE predates this PR and the gate's
VOICE branch ignores media_types, so a NEW connector against an OLD
gateway ALREADY fires STT. That is desirable, but it is not "unchanged"
— the PR body's rollout matrix said otherwise and is corrected.

* fix(relay): send a User-Agent on relay media requests (Discord CDN 403)

Discord's CDN rejects urllib's default "Python-urllib/x.y" User-Agent
with HTTP 403, and RelayMediaClient never set one. Every Discord CDN
pass-through download therefore failed; _localize_inbound_media then
kept the raw URL (its "a public URL still has value" branch), and the
consumer tried to open a URL as a FILE PATH:

  WARNING gateway.relay.media: relay media download failed for
    https://cdn.discordapp.com/...voice-message.ogg: HTTP Error 403
  INFO gateway.run: Voice transcription failed for https://cdn.discord...
    : Audio file not found: https://cdn.discordapp.com/...

This killed ALL Discord relay media inbound — voice notes, images and
documents alike — not just the voice lane. Telegram/WhatsApp were
unaffected because their media is connector-re-hosted (/relay/media/{id},
fetched from our own host) and localizes to real /tmp paths.

Reproduced from a clean shell against a live CDN URL:
  curl (own UA)                -> 200
  urllib, no UA                -> 403 Forbidden
  urllib + descriptive UA      -> 200, 14583 bytes, OggS magic

Fix: a module-level _MEDIA_USER_AGENT sent on both download() and
upload(). upload() only ever targets our own connector so it was not
broken, but a single client should identify itself consistently.

Validated on staging: hot-patched hermes-agent-stg-test-6698, restarted
the gateway service, and Ben's Discord voice note transcribed
successfully — zero new 403s and zero new transcription failures after
the patch (last 403 predates it).

Test is mutation-verified: removing the UA from download() turns it RED
while the other five media tests stay green.

* fix(relay): keep url↔mime pairing through media localization

Addresses a blocking review finding on my own change: mapping media[]
into media_types created a POSITIONAL contract that the rest of the
inbound path then broke.

1. _localize_inbound_media (adapter.py) filtered media_urls without
   filtering media_types. Dropping a dead connector re-host is a NORMAL
   best-effort path, so every surviving attachment inherited its
   neighbour's mime. Reproduced through the real functions:

     before urls  [.../relay/media/dead, .../kept.png]
            types [application/pdf, image/png]
     after  urls  [.../kept.png]
            types [application/pdf, image/png]   <-- PNG reads as PDF
     _event_media_is_image(ev, 0) -> False

   The loop now carries (url, mime) as PAIRS, so a dropped URL drops its
   mime with it.

2. _media_types_from_wire compared LENGTHS only, which is not alignment:
   equal-length-but-reordered wire fields were accepted and paired
   wrongly, and an absent media_urls skipped the check entirely while
   still emitting types. Resolution is now BY URL (url -> mime lookup
   over media_urls); an unmatched URL degrades to "" and falls back to
   message-level classification.

Tests: 4 new cases driving the real chain (wire parse -> localization ->
run.py classifier), incl. the dropped-first-attachment case the existing
localization test could not catch (it builds events without
media_types). The obsolete length-mismatch test now asserts the stronger
by-url guarantee. Both fixes mutation-verified: reinstating the URL-only
filter fails 1 test, reverting to positional resolution fails 3.

Relay suite 258 passed; media/voice/stt selection 685 passed; ruff clean;
cross-repo integration payload re-verified.

* fix(relay): media_types is always one slot per media_url

Self-review after two review rounds flagged this bug class in adjacent
seams: I checked the function I edited, not every consumer of the
parallel arrays I created. Grepping ALL writers found a third instance.

merge_pending_message_event (gateway/platforms/base.py:2725-2735)
EXTENDS media_urls and media_types together when a second media message
merges into a pending one. My mapping could emit a POPULATED media_urls
with an EMPTY media_types (an older connector sends media_urls but no
media[]), so extend() concatenated lists of different lengths:

  A urls [old1.png, old2.png]  types []
  B urls [new.pdf]             types [application/pdf]
  merged urls  [old1.png, old2.png, new.pdf]
         types [application/pdf]
    -> old1.png reads as application/pdf; the real PDF gets ''

Fix: media_types is now ALWAYS len(media_urls), padded with '' — the
url-keyed lookup runs even when media[] is absent, and the localizer
rewrites the list unconditionally (no  short-circuit that
could leave a stale/short list behind).

Tests: 4 new cases — padding with no media[], the merge shift above
driven through the real merge_pending_message_event, localization
preserving the invariant while dropping an entry, and normalization of
a short/empty media_types arriving from a non-wire source. All
mutation-verified: removing the padding fails 4; restoring the
 guard fails 1.

Relay 262 passed; media/voice/stt selection 689 passed; ruff clean;
cross-repo integration payload re-verified.
2026-08-27 15:07:38 +10:00

232 lines
8.5 KiB
Python

"""Relay Phase 2 media tests — send_media egress lanes + inbound media localization.
Covers:
- the five ``send_*`` overrides route through ONE ``send_media`` op with the
right ``media_kind`` and honor op-level capability gating (a connector not
advertising ``send_media`` falls back to the base-class behaviour);
- local-path sources upload through the RelayMediaClient first (the
connector cannot reach our filesystem) and public URLs pass through;
- a connector decline / failed upload degrades to the pre-media fallback;
- inbound ``media_urls`` are localized to temp paths (re-hosts downloaded
with the per-gateway bearer; dead re-host refs dropped; public URLs kept
when no client is available);
- the RelayMediaClient URL derivation + auth header shape.
"""
from __future__ import annotations
from pathlib import Path
from typing import Optional
import pytest
from gateway.config import PlatformConfig
from gateway.relay.adapter import RelayAdapter
from gateway.relay.descriptor import CONTRACT_VERSION, CapabilityDescriptor
from gateway.relay.media import RelayMediaClient, media_base_url
from tests.gateway.relay.stub_connector import StubConnector
def make_desc(**kw) -> CapabilityDescriptor:
base = dict(
contract_version=CONTRACT_VERSION,
platform="telegram",
label="Telegram",
max_message_length=4096,
supports_draft_streaming=False,
supports_edit=True,
supports_threads=True,
markdown_dialect="markdown_v2",
len_unit="utf16",
supported_ops=(
"send",
"edit",
"typing",
"get_chat_info",
"send_media",
),
)
base.update(kw)
return CapabilityDescriptor(**base)
class FakeMediaClient:
"""In-memory stand-in for RelayMediaClient (no HTTP)."""
def __init__(self) -> None:
self.enabled = True
self.uploads: list[tuple[str, Optional[str]]] = []
self.downloads: list[str] = []
self.upload_result: Optional[str] = "https://conn.example/relay/media/aa11"
self.download_result: Optional[str] = "/tmp/relay_media_fake.png"
async def upload(self, file_path, *, mime=None, filename=None):
self.uploads.append((str(file_path), filename))
return self.upload_result
async def download(self, url, *, suggested_name=None):
self.downloads.append(url)
return self.download_result
def is_relay_media_url(self, url: str) -> bool:
return "/relay/media/" in (url or "")
def _adapter(**desc_kw) -> tuple[RelayAdapter, StubConnector, FakeMediaClient]:
stub = StubConnector(make_desc(**desc_kw))
adapter = RelayAdapter(PlatformConfig(), make_desc(**desc_kw), transport=stub)
fake = FakeMediaClient()
adapter._media_client = fake # bypass env-derived construction
return adapter, stub, fake
# ── egress: the five overrides ───────────────────────────────────────────
@pytest.mark.asyncio
async def test_send_image_url_passes_through_without_upload():
adapter, stub, fake = _adapter()
result = await adapter.send_image(
"chat1", "https://fal.media/x.png", caption="a pic", reply_to="m9"
)
assert result.success is True
assert result.message_id == "md1"
assert fake.uploads == [] # public URL → no upload leg
action = stub.sent[-1]
assert action["op"] == "send_media"
assert action["media_kind"] == "image"
assert action["source_url"] == "https://fal.media/x.png"
assert action["content"] == "a pic"
assert action["reply_to"] == "m9"
@pytest.mark.asyncio
async def test_local_path_lanes_upload_first(tmp_path: Path):
adapter, stub, fake = _adapter()
f = tmp_path / "clip.ogg"
f.write_bytes(b"oggbytes")
result = await adapter.send_voice("chat1", str(f), caption="listen")
assert result.success is True
assert fake.uploads == [(str(f), None)]
action = stub.sent[-1]
assert action["op"] == "send_media"
assert action["media_kind"] == "voice"
# The wire carries the RE-HOST reference, never the local path.
assert action["source_url"] == fake.upload_result
assert str(f) not in str(action)
@pytest.mark.asyncio
async def test_op_gating_falls_back_when_not_advertised(tmp_path: Path):
# Connector advertises only the legacy ops — send_media must never hit the wire.
adapter, stub, fake = _adapter(
supported_ops=("send", "edit", "typing", "get_chat_info")
)
result = await adapter.send_image("chat1", "https://x.io/a.png", caption="hi")
# Base-class fallback: caption + URL as a text send.
assert result.success is True
ops = [a["op"] for a in stub.sent]
assert "send_media" not in ops
assert ops[-1] == "send"
assert "https://x.io/a.png" in stub.sent[-1]["content"]
# ── inbound localization ─────────────────────────────────────────────────
def _make_event(media_urls):
from gateway.platforms.base import MessageEvent, MessageType
from gateway.session import SessionSource
return MessageEvent(
text="look",
message_type=MessageType.TEXT,
source=SessionSource(
platform="telegram", chat_id="c1", chat_type="dm", user_id="u1"
),
media_urls=list(media_urls),
)
@pytest.mark.asyncio
async def test_inbound_without_client_keeps_public_drops_rehost():
adapter, _stub, _fake = _adapter()
adapter._media_client = None
adapter._get_media_client = lambda: None # type: ignore[method-assign]
event = _make_event(
[
"https://conn.example/relay/media/deadbeef",
"https://cdn.discordapp.com/attachments/a/b.png",
]
)
await adapter._localize_inbound_media(event)
assert event.media_urls == ["https://cdn.discordapp.com/attachments/a/b.png"]
# ── RelayMediaClient unit surface ────────────────────────────────────────
@pytest.mark.asyncio
async def test_client_upload_rejects_oversize_and_missing(tmp_path: Path):
c = RelayMediaClient("https://c.example", "gw1", "sec")
# Missing file → None (no network attempted).
assert await c.upload(str(tmp_path / "nope.bin")) is None
# Empty file → None.
empty = tmp_path / "empty.bin"
empty.write_bytes(b"")
assert await c.upload(str(empty)) is None
@pytest.mark.asyncio
async def test_download_sends_a_user_agent_on_every_request():
"""Discord's CDN 403s the default ``Python-urllib/x.y`` User-Agent.
Live-verified on staging 2026-08-26: every Discord CDN pass-through
download failed with ``HTTP Error 403: Forbidden``, the localizer then kept
the raw URL, and the transcriber tried to open a URL as a file path
("Audio file not found") — so voice notes, images and documents were ALL
silently dead on the Discord relay lane. Reproduced from a clean shell:
urllib with no UA → 403; the same URL with any descriptive UA → 200.
Assert the header on BOTH url classes (public pass-through and
bearer-authenticated re-host), because they take different header paths.
"""
seen: list[dict] = []
class _Resp:
headers = {"Content-Type": "audio/ogg", "Content-Length": "4"}
def read(self, *_a):
return b"OggS"
def __enter__(self):
return self
def __exit__(self, *_a):
return False
def _fake_urlopen(req, timeout=None): # noqa: ARG001
seen.append(dict(req.headers))
return _Resp()
import urllib.request as _ur
orig = _ur.urlopen
_ur.urlopen = _fake_urlopen # type: ignore[assignment]
try:
c = RelayMediaClient("https://conn.example", "gw1", "sec")
assert await c.download("https://cdn.discordapp.com/attachments/1/2/v.ogg")
assert await c.download("https://conn.example/relay/media/deadbeef")
finally:
_ur.urlopen = orig # type: ignore[assignment]
assert len(seen) == 2
for headers in seen:
# urllib title-cases header keys on Request.
ua = headers.get("User-agent") or headers.get("User-Agent")
assert ua, f"no User-Agent sent; urllib would default to Python-urllib (403s on Discord CDN): {headers}"
assert "python-urllib" not in ua.lower()
# The re-host request must still carry its bearer (no regression).
rehost_headers = seen[1]
assert (rehost_headers.get("Authorization") or "").startswith("Bearer ")