feat(telemetry): transmit the stable install_id as-is
Product-owner decision, 2026-08-27: the analytical need is stable cross-window identity (retention curves, longitudinal install behaviour), which the rotating pseudonym destroyed by design. The feature has not shipped - zero consented users, zero production transmissions - so identity semantics can change without breaking any promise made to a user; existing (dev-only) consent windows carry forward unchanged. Removed in full rather than weakened in place: - shared_metrics_identity.py (salt generation/rotation, HMAC-SHA256 derivation, payload substitution) and its 19-test file. - The sender's derivation step. _freeze_identity keeps its validation role (unreadable/non-object/id-less payloads still reject rather than block the queue) and now records the raw install_id in sent_install_id; _body rewrites the payload's install_id from that frozen column, keeping byte-identical resends anchored to one recorded value. Consent surface updated in the same change: the setup wizard now states plainly that packages carry the stable profile-scoped install ID (a random UUID, no personal information, reset by deleting the shared-metrics directory). No consent was ever collected under the old wording in any shipped build. Docs A.2/A.3 rewritten as decision records rather than silently edited: A.2 records what is transmitted now and states the consequences plainly (indefinite cross-package correlation is the designed behaviour); A.3 records why rotation existed and why its removal was accepted. The main-body "must not reuse the persistent local identifier by default" escape hatch is exercised, not deleted: that paragraph required exactly this product decision, which has now been made. A.6's deletion note updated: install_id is now itself the lookup key, so a future delete-on-request needs only a service-side API, not a mapping. Tests: the two privacy assertions invert deliberately (test_the_stable_install_id_is_transmitted_as_is and the e2e wire variant); freezing/byte-identical-retry coverage unchanged. Staging E2E script now asserts transmitted == install_id. 258 targeted tests pass; ruff + footguns clean; both staging E2E harnesses green with the raw id observed on the wire (202s).
This commit is contained in:
@@ -230,17 +230,18 @@ packages from that profile and can therefore link those local packages.
|
||||
Deleting `$HERMES_HOME/telemetry/shared_metrics` resets the identifier together
|
||||
with all aggregates and package files.
|
||||
|
||||
Remote delivery is opt-in and off by default. A remote exporter must not reuse
|
||||
the persistent local identifier by default. It requires a separate product and
|
||||
privacy decision covering consent, identity scope, rotation or keyed
|
||||
pseudonymization, reset behavior, retention, and deletion.
|
||||
Remote delivery is opt-in and off by default. Reusing the persistent local
|
||||
identifier remotely required a separate product and privacy decision covering
|
||||
consent, identity scope, reset behavior, retention, and deletion — that
|
||||
decision has been made.
|
||||
|
||||
> Those decisions are recorded in
|
||||
> [Appendix A](#appendix-a-remote-exporter-decisions-phase-2), and the exporter
|
||||
> implementing them has shipped. Collection alone still transmits nothing: the
|
||||
> sender runs only when `telemetry.shared_metrics.send` is also true, and it
|
||||
> transmits a rotating HMAC of the install identity rather than the identifier
|
||||
> itself.
|
||||
> sender runs only when `telemetry.shared_metrics.send` is also true. Each
|
||||
> transmitted package carries the stable `install_id` as-is (product decision,
|
||||
> 2026-08-27 — see A.2 for the record, including the superseded
|
||||
> HMAC-pseudonym design).
|
||||
|
||||
The install identity is scoped to one `HERMES_HOME`. To reset it, stop Hermes
|
||||
processes and remove `$HERMES_HOME/telemetry/shared_metrics`. This deliberately
|
||||
@@ -327,60 +328,66 @@ Local history can be up to 30 days old, and that data was collected under a
|
||||
promise that nothing is uploaded. Honouring consent forward-only costs at most
|
||||
30 days of backlog we never had permission to send.
|
||||
|
||||
### A.2 Identity scope — the transmitted identifier is derived, not the local one
|
||||
### A.2 Identity scope — the stable install_id is transmitted as-is
|
||||
|
||||
`install_id` is the persistent profile-scoped identifier described above. It is
|
||||
**not transmitted**. Each package sent carries a derived value instead:
|
||||
**Decision record.** The original design of this exporter (and revisions 1–8
|
||||
of this appendix) transmitted a keyed pseudonym instead of the identifier:
|
||||
`HMAC-SHA256(key = locally-held rotating salt, message = install_id)`, with
|
||||
the salt rotating every 30 days. On **2026-08-27**, before the feature
|
||||
shipped (zero consented users, zero production transmissions), the product
|
||||
owner decided the analytical need is a **stable cross-window identity** —
|
||||
retention curves, longitudinal install behaviour — which rotation by design
|
||||
destroys. The pseudonymization layer was removed in full rather than
|
||||
weakened in place.
|
||||
|
||||
```text
|
||||
transmitted_id = HMAC-SHA256(key = rotation_salt, message = install_id)
|
||||
```
|
||||
What is transmitted now:
|
||||
|
||||
- `rotation_salt` is random, generated locally, and never leaves the machine.
|
||||
- The derivation is one-way: the service cannot recover `install_id`.
|
||||
- Within a rotation window, packages from one profile correlate — so distinct
|
||||
installs remain countable, which is the primary analytical question.
|
||||
- Across windows, they do not.
|
||||
- Each package carries `install_id` verbatim: the persistent, profile-scoped
|
||||
random UUID described above.
|
||||
- It is generated locally (`uuid4`), contains no hardware, account, user, or
|
||||
machine-derived information, and identifies a *profile*, not a person.
|
||||
- It is stable until the user deletes the shared-metrics directory, which
|
||||
regenerates it (see A.4).
|
||||
|
||||
This satisfies "must not reuse the persistent local identifier by default"
|
||||
while keeping the data useful. Stripping the identifier entirely was rejected
|
||||
because "how many installs are reporting" is the first question the data must
|
||||
answer; sending `install_id` unchanged was rejected because it contradicts the
|
||||
commitment made above.
|
||||
Consequences stated plainly rather than papered over:
|
||||
|
||||
**Byte-identical resends still hold.** The derived value is computed **once**,
|
||||
when the package is first prepared for sending, and stored alongside the
|
||||
package (the derived id only — not a second copy of the payload, which is
|
||||
recomputed deterministically from the stored package). A retry therefore
|
||||
rebuilds identical bytes even if the salt rotated in between. The contract
|
||||
requires this: resending a `package_id` with different content is undefined
|
||||
behaviour.
|
||||
- Packages from one profile correlate **indefinitely**, not per-window.
|
||||
Long-term linkability of one install's daily envelope sequence is now the
|
||||
designed behaviour, not a residue.
|
||||
- The A.3 residue analysis of the old design (stable `resource` tuple +
|
||||
contiguous periods bridging rotation windows) is moot — there is no window
|
||||
boundary left to bridge.
|
||||
- The setup wizard's consent language states this identity model explicitly;
|
||||
it was updated in the same change that removed the derivation, so no
|
||||
consent was ever collected under the old wording in any shipped build.
|
||||
|
||||
### A.3 Rotation
|
||||
**Byte-identical resends still hold.** The transmitted id is recorded on the
|
||||
row (`sent_install_id`) when the package is first prepared, and the wire body
|
||||
is always rebuilt from that recorded value, so a retry rebuilds identical
|
||||
bytes. The contract requires this: resending a `package_id` with different
|
||||
content is undefined behaviour. (With a stable id the recorded copy is no
|
||||
longer load-bearing against rotation — it remains as the audit column and as
|
||||
cheap insurance against any future change to identity semantics.)
|
||||
|
||||
`rotation_salt` rotates on a fixed schedule (default: every 30 days, aligned to
|
||||
local history retention). Rotation only affects packages prepared after it;
|
||||
already-prepared packages keep their derived value so retries stay
|
||||
byte-identical.
|
||||
### A.3 Rotation — removed (decision record)
|
||||
|
||||
Rotation bounds long-term linkability without destroying short-term cohort
|
||||
analysis. A profile is one identity for the length of a window, and an
|
||||
unrelated identity after it.
|
||||
Salt rotation was deleted together with the derivation (product decision,
|
||||
2026-08-27). This section is retained as a record of what the earlier design
|
||||
did and why the removal was accepted:
|
||||
|
||||
**What rotation does not bound.** The identifier changes; the rest of the
|
||||
envelope does not. `resource` (`os_family`, `architecture`, `install_method`,
|
||||
`hermes_version`) is stable and low-entropy, and `period_start` /
|
||||
`period_end` are contiguous across a rotation boundary. For a common
|
||||
configuration this is no help to an observer — measured against the 11 real
|
||||
packages in a development outbox, every one shares the same
|
||||
`arm64 / macos / git` tuple. For a **rare** configuration it is a plausible
|
||||
re-identification aid: an unusual architecture or install method, combined
|
||||
with an uninterrupted daily period sequence, can bridge two windows. The
|
||||
claim this design makes is therefore "rotation raises the cost of long-term
|
||||
correlation", not "rotation makes it impossible". Narrowing that residue
|
||||
would mean coarsening `resource` or jittering period boundaries, and neither
|
||||
is worth the analytical loss today — but it should be a conscious decision,
|
||||
not an unexamined one.
|
||||
- Rotation existed to bound long-term linkability: one identity per 30-day
|
||||
window, unrelated identities across windows.
|
||||
- The documented residue (see git history for the full analysis): the
|
||||
envelope's stable, low-entropy `resource` tuple plus contiguous daily
|
||||
periods could plausibly bridge windows for rare configurations anyway, so
|
||||
the boundary was a cost-raiser, not a wall.
|
||||
- The product need that killed it: cross-window continuity is precisely what
|
||||
retention analysis requires. A boundary that mostly inconveniences honest
|
||||
analysis while only raising costs for a determined correlator was judged
|
||||
the wrong trade once stable identity became a requirement.
|
||||
|
||||
There is no salt in the store, no rotation schedule, and no derived
|
||||
identifier anywhere in the pipeline.
|
||||
|
||||
### A.4 Reset behavior
|
||||
|
||||
@@ -388,11 +395,11 @@ Removing `$HERMES_HOME/telemetry/shared_metrics` still resets local identity,
|
||||
aggregates, and package files, exactly as documented above. Two honest
|
||||
qualifications now apply:
|
||||
|
||||
- Reset also discards `rotation_salt`, so subsequent packages derive a **new**
|
||||
transmitted identity. Local reset does give a new remote identity.
|
||||
- Reset regenerates `install_id`, so subsequent packages transmit a **new**
|
||||
identity. Local reset does give a new remote identity.
|
||||
- Reset **cannot unsend**. Packages already transmitted remain in the ingest
|
||||
service's storage under their derived identifier. There is no read-back or
|
||||
delete API in the v1 contract.
|
||||
service's storage under the identifier they were sent with. There is no
|
||||
read-back or delete API in the v1 contract.
|
||||
|
||||
Setting `send: false` stops transmission immediately: consent is re-read
|
||||
before every package, so a pass already in flight stops after the package it
|
||||
@@ -439,13 +446,13 @@ invent one. What a user can do:
|
||||
|---|---|
|
||||
| `send: false` | No further packages leave the machine |
|
||||
| `enabled: false` | Collection stops; existing local state remains |
|
||||
| Remove `.../shared_metrics` | Local identity, aggregates, and files reset; future sends use a new derived identity |
|
||||
| Remove `.../shared_metrics` | Local identity, aggregates, and files reset; future sends use a new install_id |
|
||||
| Delete already-sent data | Not self-service — requires an operator acting on the S3 bucket |
|
||||
|
||||
If a deletion-on-request obligation is ever taken on, it needs a lookup path
|
||||
from a user to their derived identifiers. That is deliberately **not** built:
|
||||
it would require retaining the mapping this design exists to avoid. Any such
|
||||
change is a new product decision, not an implementation detail.
|
||||
If a deletion-on-request obligation is ever taken on, the lookup path is now
|
||||
direct: the user's `install_id` (readable from their local store) is the key
|
||||
their data is stored under. Building the service-side delete API remains a
|
||||
new product decision, not an implementation detail.
|
||||
|
||||
### A.7 What the outbox directory is
|
||||
|
||||
@@ -464,7 +471,8 @@ state they were promised. Send state lives in new columns on the
|
||||
|
||||
### A.8 Scope note
|
||||
|
||||
The `install_id` field inside the package body is what gets replaced by the
|
||||
derived value. No other payload field changes, nothing is added, and the
|
||||
service treats the whole body as opaque. Payload schema evolution therefore
|
||||
stays a sender-side concern, as before.
|
||||
The `install_id` field inside the package body is transmitted as the
|
||||
generator wrote it (rewritten from the row's frozen `sent_install_id`, which
|
||||
records the same value). No other payload field changes, nothing is added,
|
||||
and the service treats the whole body as opaque. Payload schema evolution
|
||||
therefore stays a sender-side concern, as before.
|
||||
|
||||
Reference in New Issue
Block a user