The slowest green leg ever recorded is 29 minutes; every cap hit in the
suite's history was a hang, never work. Caps were linux 75 / macos 120 /
windows 240, so a wedged leg burned up to 4 hours of runner time to
report what its log showed in the first minutes. 60 minutes covers the
slowest leg plus cold-cache variance, and every driver-internal bound
(dmg install 45m, AHK 50m, updater wait) still fires before the job cap
in any single-hang scenario, keeping failure diagnostics specific.
The detached-updater wait drops 90m -> 35m on the same evidence: a
working updater finishes far inside 35m; a wedged one never finishes at
any bound, and the longer wait only delayed the report by an hour.
A dmg user can also update from the terminal (hermes update) or by
re-running the install one-liner; the dmg arm only routed the two
app-button methods, leaving three declared cells as permanent TODOs.
Port the POSIX driver's method blocks into the macos driver's update
phase (hermes-update with the --yes probe, installer re-run with per-ref
flag probing, the +desktop built-app assert) and open the workflow gate.
The dmg bootstrap driver also learns to recover instead of waiting out
its bound when a stage fails: the bootstrap parks on an error screen
with a Retry button (seen live: HTTP 429 downloading install.sh under
full-matrix runner load), so the driver watches the bootstrap log for
real failure shapes, requires two consecutive error probes before
retargeting the click at Retry's measured position, unlatches when the
log goes healthy, and gives up with the true cause after three retries.
Driver rule learned three times in this suite (lsof +D, git show and
find piped to grep -q): under set -euo pipefail, never feed grep -q
from a pipe... grep exits at first match, the producer takes SIGPIPE,
and a TRUE condition reads as failure. Capture to a variable or test
paths directly.
Verified end to end: run 33506177130, macos slice 13/13 green.
Windows transcripts were ZERO bytes: ts-prefix.ps1 formatted with
{0:D2}, but Floor() returns a double and the D specifier is
integer-only - it threw per line, and under the driver's relaxed EAP
every line errored into the void. {0:00} fixes it (custom numeric
format works on doubles). Reproduced the exact pipeline locally
(empty file + Format specifier invalid), verified the fix produces
prefixed merged stdout+stderr with exit code intact. That is also
why the log timeline never auto-synced: there was nothing in the
files to sync.
The GitHub artifact URL 307s to /suites/... server-side and strips
the ?zip= query param. The player now reads the zip URL from a
#zip= HASH param (client-side, survives the redirect) with ?zip=
as fallback; the hash path was verified in a real browser against a
real leg zip (auto-fetch + boot).
Per ethie's design, one player artifact for the whole run: new
leg-player job uploads playback.html (archive:false) before the
matrix legs, the report job needs it, and each ran cell gets TWO
links - 📼 to the player with #zip=<that leg's logs zip> and ⬇️ to
the raw zip. Per-leg player uploads removed from all three run
workflows.
Each leg uploads playback.html as a single-file artifact (archive:
false) before the driver runs, so it exists even on failure. The
results chart now links every leg that RAN (pass or fail, not skip)
to its player with ?zip= pointing at that leg's logs artifact.
Leg<->artifact mapping: the generator mints a leg_id per matrix entry
(sanitized matrix name, exported legId()), every run workflow names
its artifacts install-e2e-{player,logs}-<leg-id>, and the report job
feeds the run's artifact name->id list to the results renderer, which
rebuilds the leg id from the parsed job name. GitHub does not link
jobs to artifacts, so the deterministic name is the join key.
Empirical finding: GitHub artifact downloads are auth-gated (the
download URL 307s to /suites/... which is 404 anonymous), so a
locally-opened player page cannot fetch the zip cross-origin. The
player now degrades gracefully: ?zip= fetch failure renders a real
download link for the zip (a normal click carries the user's session)
plus a drag-and-drop / file-picker path, and no-param opens as a pure
drop target. Verified in a real browser against a real artifact URL.
Verified: generator emits leg_id, results renderer emits
✅/❌ [📼](...?zip=...) only on ran cells, npm run check PASS,
install tests 36/36, strict tsc PASS, actionlint x4 PASS.
A static single-file player (tests/install/e2e-assets/playback.html):
?zip=<artifact zip url> unzips in-browser (JSZip), plays the screen
recording with a timer pinned top-left, and renders every *.log with
video<->log sync: the video follows the driver's transcript, clicking
a log line seeks the video. A sync-offset slider aligns the recording
start (ffmpeg comes up first) with the driver's relative clock.
Sync axis: drivers now prefix every transcript line with [+MM:SS]
relative to driver start (ts-prefix.sh / ts-prefix.ps1, pipe-safe
under pipefail / relaxed EAP). Browsers cannot play Matroska, so each
leg remuxes recording.mkv -> recording.mp4 (-c copy, no re-encode)
before the artifact upload, on all three OSes.
Verified end-to-end in a real browser against a generated artifact
zip: zip load, mp4 playback, timer, tab switching, follow-sync at
t=6/t=12, click-to-seek, autoplay policy (expected NotAllowedError on
synthetic play; real clicks fine).
Also fixes the shim fail message's dead variable ( ->
observed_git_url) in both posix drivers.
macos gains the desktop-installer@latest install method: the website's
Hermes-Setup.dmg (verified live), mounted with hdiutil and its app
binary run DIRECTLY - an open-launched app inherits none of the git
redirect env, so direct exec is what keeps the isolation honest while
staying the same binary and first-launch flow.
install-e2e-macos-run.yml takes the windows shape: one workflow, one
inner job per driver arm, native skips. Arm 1 delegates script installs
to the shared OS-agnostic run workflow; arm 2 stages, installs from the
dmg, and drives both app-update methods through launch-from-spec.mjs -
open-app-update launches the installed .app (the double-click surface,
env via Playwright), hermes-desktop-app-update captures the product's
own hermes desktop spawn. Both end on sha asserts, never version
strings.
The fake Internet (bubblewrap + slirp4netns + MITM proxy +
upload-pack shim, 883 lines across dev-sandbox.sh, stage2-run.sh,
proxy.py, ssh-shim.sh, openssl.cnf, install-update-e2e.sh) existed to
isolate install.sh's network. The GIT_CONFIG_GLOBAL insteadOf redirect
the windows driver introduced does the same job with a gitconfig file
and works on any OS, so:
* install-e2e-run.yml now runs tests/install/installer-script-e2e.sh
directly on the bare runner - no sandbox deps, no userns sysctls -
and takes a runner input;
* the macos matrix calls the SAME workflow on macos-latest, deleting
install-e2e-macos-run.yml: installer-script -> installer-script /
hermes-update flip from grey to live, app-update pairs stay TODO
inside the shared gate;
* install.sh is no longer curl'd through a fake CA - each leg runs
the copy from the ref a user of that version actually executed;
* scripts/dev-sandbox.sh becomes the minimal isolation sandbox from
ab6b9492f (separate HERMES_HOME / Electron userData / app name,
same CLI surface: --persistent, --from, --delete), keeping its
.hermes-sandbox dir name so gitignore and docs hold;
* nix/sandbox.nix drops the bwrap/proxy closure and keeps only the
Electron runtime LD_LIBRARY_PATH the desktop app needs.
Verified: nix build .#sandbox + smoke run (isolated HERMES_HOME
created, ephemeral cleanup), shellcheck/bash -n on both scripts,
actionlint on all three workflows, and the new driver ran the full
v0.20.2 -> HEAD hermes-update pass locally before this commit.
Per review the unions were overcomplicated. Install methods are now
just: installer-script (the platform one-liner - curl | bash on
linux/macos, irm | iex on windows), desktop-installer, and
packaged-app (declared, unused). Update methods are every install
method (re-run it over the existing install) plus hermes-update and
app-update. desktop-installer-rerun, desktop-app, curl-bash, and
irm-iex are gone as ids; the windows driver's ValidateSet, switch
arms, and both run workflows' gates renamed to match. tsc --checkJs
clean; generator output re-verified (4 linux / 16 windows / 6 macos
legs for 2 tags).
Revert the hardcoded 16-job experiment: the combination spec belongs
in scripts/sandbox/generate-e2e-matrix.mjs (restored), not copy-pasted
YAML blocks. What survives from the experiment:
* leg names carry everything - 'os: install -> update (tag -> HEAD)' -
generated per entry, since slash-joined names are all the graph
renders;
* pick-releases annotates each tag ({ref, desktop}) and the generator
threads tag_has_desktop onto windows entries, so the windows run
workflow still gates pre-desktop tags without a probe job;
* the per-OS run workflows are untouched: single job, static 'e2e'
name, native skip gates own all capability knowledge.
install-e2e.yml is one generate job + three per-OS matrix fanouts.
Generator shape (4/16/12 legs for 2 tags), annotation threading, and
all six error paths verified; all four workflows pass actionlint;
driver parses clean pure-ASCII.
Graph polish + one structural simplification, after the first render
of the combo-box layout:
* Leg names: every combination job's display name is now
'${{ matrix.tag.ref }} -> HEAD' - the box title (job id) already
carries os+methods, so repeating them per leg was noise. The inner
job renders as a short static 'e2e' tail (dynamic names render
unexpanded on skipped jobs, so it must stay static).
* The windows probe job is gone: pick-releases now annotates each
picked tag with whether its tree ships apps/desktop
({ref, desktop} objects in the matrix), and the windows run
workflow gates on the new tag-has-desktop boolean input directly.
One tree listing at pick time replaces N probe jobs, and the
'probe tag' noise disappears from the graph.
Annotation loop verified against the real tag set (pre/post-desktop
split lands exactly at the app's introduction); 16-combo inventory
re-asserted; all four workflows pass actionlint.
GitHub only draws matrix boxes for the PRIMARY workflow's matrices;
everything inside a called workflow flattens into slash-joined names.
The generator + per-tag sub-workflow therefore bought no structure in
the graph and hid the support matrix in a script.
Invert it: install-e2e.yml now declares one job per {os,
install-method -> update-method} combination (2 linux + 8 windows +
6 macos - same 16 the generator produced, verified by inventory
before/after), each a matrix over the picked release tags. The graph
now renders one titled box per combination whose legs read
'... from vX' - the tag axis inside the combo axis. The per-OS run
workflows are unchanged: they own capability knowledge and natively
skip unimplemented method pairs and pre-desktop tags.
install-e2e-tag.yml and generate-e2e-matrix.mjs are deleted; adding a
method is now adding one job block here, implementing one is flipping
the run workflow's gate.
Remove all capability knowledge from the combination generator: no
IMPLEMENTED table, no skipped matrix, no per-OS special cases. It now
only declares and expands - every {os, install-method, update-method}
combination is dispatched to its OS's run workflow, and each run
workflow natively skips (grey, job-level if on the method inputs) the
pairs its driver cannot run yet:
* install-e2e-run.yml gains install-method/update-method inputs,
gates on the supported pairs (curl-bash -> hermes-update/curl-bash),
and maps the method id to the sandbox script's --route internally;
* install-e2e-macos-run.yml is new - all pairs skip until a macOS
driver exists, and implementing one flips its job-level if;
* install-e2e-windows-run.yml already worked this way;
* install-e2e-skip.yml is deleted - nothing special-cases macOS
anymore, so the tag workflow is three identical OS fanouts.
Structure is now uniformly matrix(tag) -> matrix(combination) ->
run-or-skip, with capability knowledge living only next to each
driver. Generator shape/route filters/error paths re-verified; all
five workflows pass actionlint.
yoniebans