Ben Barclay
613849c190
Fourth independent review. Two more consent leaks, both reproduced through the real relay entry point before and after the fix. Both are failures of my own round-3 fix, which recorded revocation in the wrong place. BLOCKER 1 - revoking while idle recorded nothing. _record_revocation lived inside send_pending's loop, but _send_exported_packages returns early when send is false, before a sender is ever constructed. The dominant case is a user turning sending off while no pass is running, so the loop that was meant to observe the revocation could never run. Reproduced: 6 periods collected during a refused window were transmitted on re-enable. The window now closes on the observed config EDGE, before the early return. Last-seen send state is persisted because each hook fires in a fresh process, so a true->false transition is only visible by comparison. The rising edge also opens the window explicitly: the sender only runs when there is something to send, so a user who opts in and out before any package exists would otherwise have no window for record_revoked to close. BLOCKER 2 - turning COLLECTION off never recorded revocation. The not-enabled branch in setup.py force-set send=false and returned without calling _record_send_consent_change, so `hermes tools` -> disable shared metrics silently dropped consent while leaving the window open. Same retroactive release on re-enable. Both consent surfaces now record, and setup keeps the relay's edge detector in step. Also, from the same review's mutation sweep: - the scheme check is now pinned as an allowlist. Replacing the http test with `if True` survived the entire suite, because every non-http case targeted a REMOTE host where the loopback branch rejects anyway. Only a non-http scheme on loopback distinguishes the two. Shipped behaviour was already correct; nothing guarded it. - A.3 no longer claims rotation bounds long-term linkability outright. Measured against 11 real packages: resource is a stable low-entropy tuple and periods are contiguous across a rotation, so for a RARE configuration those can bridge windows. The honest claim is that rotation raises the cost, not that it makes correlation impossible. Two mutants are documented as unkillable rather than papered over with tests that only appear to cover them: the _defer clamp is unreachable from any current caller, and widening the falling-edge check to an unconditional else is behaviourally equivalent because record_revoked is idempotent and no-ops without an open window. An earlier version of the anti-spurious-revocation test could not fail either - it used a never-consented store, where record_revoked no-ops regardless. Rewritten to opt in, revoke, re-enable, and then assert that a steady enabled state does not re-close the reopened window. 259 tests pass. Staging E2E re-run: both packages 202.
83 lines
2.8 KiB
Python
83 lines
2.8 KiB
Python
"""Tests for shared-metrics configuration discovery and setup."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
|
|
from hermes_cli.config import DEFAULT_CONFIG
|
|
from hermes_cli.setup import setup_telemetry
|
|
from hermes_cli.subcommands.setup import build_setup_parser
|
|
|
|
|
|
def test_shared_metrics_are_registered_disabled_by_default():
|
|
assert DEFAULT_CONFIG["telemetry"]["shared_metrics"]["enabled"] is False
|
|
|
|
|
|
def test_setup_telemetry_enables_shared_metrics(monkeypatch):
|
|
config = {}
|
|
monkeypatch.setattr(
|
|
"hermes_cli.setup.prompt_yes_no",
|
|
lambda _question, default: not default,
|
|
)
|
|
|
|
setup_telemetry(config)
|
|
|
|
assert config["telemetry"]["shared_metrics"]["enabled"] is True
|
|
|
|
|
|
def test_disabling_collection_closes_the_send_consent_window(monkeypatch, tmp_path):
|
|
"""`hermes tools` -> disable shared metrics must withdraw send consent.
|
|
|
|
The not-enabled branch returned early without recording anything, so the
|
|
consent window stayed open and re-enabling later would release every
|
|
package collected in between.
|
|
"""
|
|
from hermes_cli.observability.shared_metrics import SharedMetricsStore
|
|
from hermes_cli.observability.shared_metrics_sender import SEND_REVOKED_KEY
|
|
|
|
store = SharedMetricsStore(
|
|
database_path=tmp_path / "m.db", outbox_directory=tmp_path / "o"
|
|
)
|
|
monkeypatch.setattr(
|
|
"hermes_cli.observability.shared_metrics.SharedMetricsStore",
|
|
lambda *a, **k: store,
|
|
)
|
|
|
|
# The user had consented; now they turn collection off entirely.
|
|
monkeypatch.setattr(
|
|
"hermes_cli.setup.prompt_yes_no", lambda _question, default: False
|
|
)
|
|
config = {"telemetry": {"shared_metrics": {"enabled": True, "send": True}}}
|
|
# Consent was granted earlier, so a window is already open — that is
|
|
# precisely the state whose closure must be recorded.
|
|
from hermes_cli.sqlite_util import write_txn
|
|
from hermes_cli.observability.shared_metrics_sender import opt_in_period
|
|
|
|
with store._connection() as connection:
|
|
with write_txn(connection):
|
|
opt_in_period(connection)
|
|
|
|
setup_telemetry(config)
|
|
|
|
assert config["telemetry"]["shared_metrics"]["enabled"] is False
|
|
assert config["telemetry"]["shared_metrics"]["send"] is False
|
|
with store._connection() as connection:
|
|
row = connection.execute(
|
|
"SELECT value FROM telemetry_state WHERE key = ?", (SEND_REVOKED_KEY,)
|
|
).fetchone()
|
|
assert row is not None and row[0] == "1", (
|
|
"disabling collection left the send consent window open"
|
|
)
|
|
|
|
|
|
def test_setup_parser_accepts_telemetry_section():
|
|
parser = argparse.ArgumentParser()
|
|
subparsers = parser.add_subparsers(dest="command")
|
|
handler = object()
|
|
build_setup_parser(subparsers, cmd_setup=handler)
|
|
|
|
args = parser.parse_args(["setup", "telemetry"])
|
|
|
|
assert args.section == "telemetry"
|
|
assert args.func is handler
|