Ben Barclay
49757d5e39
Independent review found the claim mechanism did not work. Reproduced against the real store: two senders POSTed the same package. The claim wrote next_attempt_at = now, but selection requires next_attempt_at <= now, so a concurrent pass matched the same row immediately. It now writes a LEASE INTO THE FUTURE (_CLAIM_LEASE_SECONDS), which is what actually excludes another pass, and expires by itself if a process dies mid-send. _mark is additionally guarded on send_state so a straggler whose lease lapsed cannot overwrite a completed send back to pending. The old concurrency test could not fail: it raised AssertionError from inside a transport, and _send_one catches every exception as a retryable transport error. It now records what the second pass saw. Also from review: - shutdown() never joined the send thread; the join was only wired into deactivate(). A short-lived CLI therefore killed an in-flight send at exit, on the only cadence this feature has. - Removed HERMES_TELEMETRY_ENDPOINT. AGENTS.md reserves HERMES_* for secrets, and a behavioural override here was a consent hazard: an inherited variable could silently redirect telemetry a user agreed to send to Nous. The staging E2E writes the endpoint into its throwaway profile instead, which also exercises the real config path. - Added the shared-metrics toggle that AGENTS.md requires as the third opt-in surface, delegating to the setup prompt so the consent rules stay in one place. - Non-429 4xx (401/403/404/413/422) are now permanent. Only 400 was, so a wrong path or oversized body retried every 15 minutes for 30 days until retention pruned it. - The opt-in day is stamped when the user consents, not on the first send pass, which silently dropped the opt-in day whenever the next export crossed midnight UTC. - gzip now uses mtime=0. The embedded timestamp made two sends of one package differ on the wire, so the 'byte-identical retry' E2E was comparing parsed bodies and could not have caught it. It now compares raw request bytes. - Reconciled the three stale claims in relay-shared-metrics.md that said no remote-delivery path exists. 233 tests pass (was 213). Staging E2E re-run through the config path: both packages 202, and the service logged both objects written to S3.
115 lines
4.2 KiB
Python
115 lines
4.2 KiB
Python
"""Configuration for shared-metrics transmission.
|
|
|
|
Collection (``telemetry.shared_metrics.enabled``) and transmission
|
|
(``telemetry.shared_metrics.send``) are separate opt-ins. See
|
|
``docs/observability/relay-shared-metrics.md`` Appendix A for the consent,
|
|
identity, rotation, retention, and deletion decisions behind this module.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from dataclasses import dataclass
|
|
from urllib.parse import urlparse
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
#: Production ingest endpoint. Overridable through config only.
|
|
#:
|
|
#: Deliberately NOT overridable by an environment variable: AGENTS.md reserves
|
|
#: HERMES_* env vars for secrets, and a behavioural override here would be a
|
|
#: consent hazard — a user who agreed to send metrics to Nous could have them
|
|
#: silently redirected to any host by an inherited variable, with nothing
|
|
#: visible in their config to show it. Tests and the staging E2E write this
|
|
#: key into a throwaway profile instead.
|
|
DEFAULT_ENDPOINT = "https://telemetry.nousresearch.com/v1/telemetry"
|
|
|
|
_LOCAL_HOSTS = frozenset({"localhost", "127.0.0.1", "::1", "[::1]"})
|
|
|
|
# Module-level latch: the enabled/send mismatch is a static misconfiguration,
|
|
# so it is reported once per process instead of on every hook fire.
|
|
_warned_send_without_collection = False
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class SendConfig:
|
|
"""Resolved transmission settings."""
|
|
|
|
#: Collection is on. Nothing is packaged or sent without it.
|
|
enabled: bool
|
|
#: Transmission is on AND permitted (that is, collection is also on).
|
|
send: bool
|
|
#: Where packages are POSTed.
|
|
endpoint: str
|
|
|
|
|
|
def _endpoint_is_safe(endpoint: str) -> bool:
|
|
"""Reject plaintext destinations unless they are loopback.
|
|
|
|
Telemetry must not leave a machine in clear text because of a typo in a
|
|
config file. Loopback stays allowed so tests can use a local HTTP server.
|
|
"""
|
|
try:
|
|
parsed = urlparse(endpoint)
|
|
except ValueError:
|
|
return False
|
|
if parsed.scheme == "https":
|
|
return True
|
|
if parsed.scheme == "http":
|
|
return (parsed.hostname or "") in _LOCAL_HOSTS
|
|
return False
|
|
|
|
|
|
def resolve_send_config(config: dict | None) -> SendConfig:
|
|
"""Resolve transmission settings from config plus the environment.
|
|
|
|
Endpoint precedence: config > production default.
|
|
|
|
``send`` is returned as False whenever transmission cannot legitimately
|
|
happen, so callers never have to re-check the combination.
|
|
"""
|
|
global _warned_send_without_collection
|
|
|
|
raw = config if isinstance(config, dict) else {}
|
|
telemetry = raw.get("telemetry")
|
|
telemetry = telemetry if isinstance(telemetry, dict) else {}
|
|
shared = telemetry.get("shared_metrics")
|
|
shared = shared if isinstance(shared, dict) else {}
|
|
|
|
enabled = shared.get("enabled") is True
|
|
send_requested = shared.get("send") is True
|
|
|
|
if send_requested and not enabled:
|
|
# Loud, not silent: the user believes telemetry is being sent, and it
|
|
# never will be. Error level, once per process.
|
|
if not _warned_send_without_collection:
|
|
_warned_send_without_collection = True
|
|
logger.error(
|
|
"telemetry.shared_metrics.send is true but "
|
|
"telemetry.shared_metrics.enabled is false — nothing is "
|
|
"collected, so nothing can be sent. Enable collection or "
|
|
"turn sending off."
|
|
)
|
|
return SendConfig(enabled=False, send=False, endpoint=DEFAULT_ENDPOINT)
|
|
|
|
endpoint = shared.get("endpoint")
|
|
if not isinstance(endpoint, str) or not endpoint.strip():
|
|
endpoint = DEFAULT_ENDPOINT
|
|
endpoint = endpoint.strip()
|
|
|
|
if send_requested and not _endpoint_is_safe(endpoint):
|
|
logger.error(
|
|
"Refusing to send shared metrics to %r: telemetry must use https "
|
|
"(or a localhost http endpoint for testing).",
|
|
endpoint,
|
|
)
|
|
return SendConfig(enabled=enabled, send=False, endpoint=endpoint)
|
|
|
|
return SendConfig(enabled=enabled, send=send_requested, endpoint=endpoint)
|
|
|
|
|
|
def reset_warning_latch_for_tests() -> None:
|
|
"""Clear the once-per-process error latch (test support only)."""
|
|
global _warned_send_without_collection
|
|
_warned_send_without_collection = False
|