Keep unknown failures red, rotate evidence per attempt, and emit receipts for signature-confirmed historical cases. Add CI-only diagnostics and an exact-tag input for the unresolved July hand-off.
Playwright 1.58 accepts a Promise-valued waitForFunction predicate before its false result. Await each zoom read explicitly; cover delayed responses and fresh-install startup.
Share zoom preparation across both launchers and stage the helper with each driver. Use the Appearance preference bridge and verify page zoom rather than display DPR.
Real Electron regressions fail with transient zoom on focus/navigation and pass with persistence. Repeated click-throughs, onboarding unit tests, E2E typecheck and lint passed. The historical onboarding timeout and full install/update matrix remain unverified.
A dmg user can also update from the terminal (hermes update) or by
re-running the install one-liner; the dmg arm only routed the two
app-button methods, leaving three declared cells as permanent TODOs.
Port the POSIX driver's method blocks into the macos driver's update
phase (hermes-update with the --yes probe, installer re-run with per-ref
flag probing, the +desktop built-app assert) and open the workflow gate.
The dmg bootstrap driver also learns to recover instead of waiting out
its bound when a stage fails: the bootstrap parks on an error screen
with a Retry button (seen live: HTTP 429 downloading install.sh under
full-matrix runner load), so the driver watches the bootstrap log for
real failure shapes, requires two consecutive error probes before
retargeting the click at Retry's measured position, unlatches when the
log goes healthy, and gives up with the true cause after three retries.
Driver rule learned three times in this suite (lsof +D, git show and
find piped to grep -q): under set -euo pipefail, never feed grep -q
from a pipe... grep exits at first match, the producer takes SIGPIPE,
and a TRUE condition reads as failure. Capture to a variable or test
paths directly.
Verified end to end: run 33506177130, macos slice 13/13 green.
Hermes-Setup is a Tauri app that boots to a setup-choice screen and waits
for a click on Install Hermes before any install work starts; run bare it
blocked until the 120-minute job cap (both dmg legs, every run). Launch it
in the background with the driver's env, read the window geometry via
System Events (position and size need no assistive grant), post a real
CGEvent click with cliclick at the button's measured position (65% of
window height; System Events' own click needs assistive access the runners
deny), then wait for the full install to land: checkout, venv console
script, AND the built Hermes.app, since the cleanup trap would otherwise
kill the installer before its desktop-build stage. Bounded at 45 minutes
with desktop screenshots on every phase and failure. Adds a macos-desktop
dispatch route so this arm iterates without the full matrix.
Verified end to end: run 33407401698, both dmg legs green (first ever),
macos slice 10/10.
Fires only on the failure path, at most every fifth iteration: what wins elementsFromPoint at the button's center, cluster and bar rects with computed z-index, the titlebar CSS vars, and window size with dpr. Makes a CI-only click interception attributable from the log alone.
On CI runners the rebuilt app cannot self-relaunch (chrome-sandbox ownership), so it parks on a reopen-to-finish overlay and never exits; a bare app.close() waits on it forever. Close with a bounded ladder instead (graceful, SIGTERM, SIGKILL), sweep the app's descendant processes so the spawned backend cannot keep writing into the install dir, then relaunch from the same captured spec and require the updated app to present a live UI. The driver gets explicit exits plus an unref'd self-deadline so it cannot outlive its own test, and the posix driver quiesces the install dir (processes whose cwd is inside it) before the head desktop smoke because the in-app update's npm is detached from the Electron process tree.
The boot-time auto-check can fail transiently and latch the error UI even though a fresh check succeeds. Click Check now whenever it is clickable, re-test Update now, 3 minute ceiling. On final failure, pull the update status over the same IPC the About panel uses so the log carries the real git error instead of the UI's generic one.
The app ships a 90% UI-zoom default, so a fresh install renders at devicePixelRatio 0.9 and Playwright's input coordinates land ~10% off target on the CI runners: clicks aimed at the titlebar settings gear hit the bar beside it, which read as a phantom overlay interception and failed every app-update leg. Set zoom to 100% through webContents after boot, retrying until dpr reads 1 because the boot path re-applies the default asynchronously.
The overlay mounts in phases (boot-progress card, then the provider picker) and the settings gear reports visible while sitting under it, so a one-shot dismiss probe and a visibility gate both race it. Alternate short-timeout dismiss clicks with settings clicks until a settings click lands; a landed click is proof the overlay is gone. Also pick the window that renders the app UI instead of trusting firstWindow(), which can return a helper webContents.
ts_prefix captured its own start per pipe, so each log's [+MM:SS]
was relative to that log's creation - the install log started at
+0:00, the app-update log at +0:00 too, and no single offset could
align all files with the recording. The drivers now stamp TS_BASE=
$SECONDS once at start and ts_prefix stamps every line relative to
it (falling back to its own start when unset): all logs in a leg
share the driver's clock, and playback.html's one offset slider
(recording start vs driver start) aligns every file at once. The ps1
twin was already driver-relative (TsPrefixStart is captured at
dot-source time, near the driver's top).
Verified: two pipes in one shell - the second starts at [+00:03],
continuing the driver clock instead of resetting to [+00:00].
The merged view interleaves every *.log in time order across the
whole leg, filename-prefixed per line (green .file span), and syncs
like any other tab: the video highlights the right file's line at the
right moment, clicking a line seeks the video. Untimed lines inherit
their file's previous timestamp so intra-file order survives the
sort; the merged tab is first and the default.
Verified in a real browser with an interleaved synthetic zip (two
logs, overlapping timelines): time-order merge, prefixes, follow-sync
at t=22 highlighting the app-update.log line, click-to-seek to t=30.
Windows transcripts were ZERO bytes: ts-prefix.ps1 formatted with
{0:D2}, but Floor() returns a double and the D specifier is
integer-only - it threw per line, and under the driver's relaxed EAP
every line errored into the void. {0:00} fixes it (custom numeric
format works on doubles). Reproduced the exact pipeline locally
(empty file + Format specifier invalid), verified the fix produces
prefixed merged stdout+stderr with exit code intact. That is also
why the log timeline never auto-synced: there was nothing in the
files to sync.
The GitHub artifact URL 307s to /suites/... server-side and strips
the ?zip= query param. The player now reads the zip URL from a
#zip= HASH param (client-side, survives the redirect) with ?zip=
as fallback; the hash path was verified in a real browser against a
real leg zip (auto-fetch + boot).
Per ethie's design, one player artifact for the whole run: new
leg-player job uploads playback.html (archive:false) before the
matrix legs, the report job needs it, and each ran cell gets TWO
links - 📼 to the player with #zip=<that leg's logs zip> and ⬇️ to
the raw zip. Per-leg player uploads removed from all three run
workflows.
Each leg uploads playback.html as a single-file artifact (archive:
false) before the driver runs, so it exists even on failure. The
results chart now links every leg that RAN (pass or fail, not skip)
to its player with ?zip= pointing at that leg's logs artifact.
Leg<->artifact mapping: the generator mints a leg_id per matrix entry
(sanitized matrix name, exported legId()), every run workflow names
its artifacts install-e2e-{player,logs}-<leg-id>, and the report job
feeds the run's artifact name->id list to the results renderer, which
rebuilds the leg id from the parsed job name. GitHub does not link
jobs to artifacts, so the deterministic name is the join key.
Empirical finding: GitHub artifact downloads are auth-gated (the
download URL 307s to /suites/... which is 404 anonymous), so a
locally-opened player page cannot fetch the zip cross-origin. The
player now degrades gracefully: ?zip= fetch failure renders a real
download link for the zip (a normal click carries the user's session)
plus a drag-and-drop / file-picker path, and no-param opens as a pure
drop target. Verified in a real browser against a real artifact URL.
Verified: generator emits leg_id, results renderer emits
✅/❌ [📼](...?zip=...) only on ran cells, npm run check PASS,
install tests 36/36, strict tsc PASS, actionlint x4 PASS.
A static single-file player (tests/install/e2e-assets/playback.html):
?zip=<artifact zip url> unzips in-browser (JSZip), plays the screen
recording with a timer pinned top-left, and renders every *.log with
video<->log sync: the video follows the driver's transcript, clicking
a log line seeks the video. A sync-offset slider aligns the recording
start (ffmpeg comes up first) with the driver's relative clock.
Sync axis: drivers now prefix every transcript line with [+MM:SS]
relative to driver start (ts-prefix.sh / ts-prefix.ps1, pipe-safe
under pipefail / relaxed EAP). Browsers cannot play Matroska, so each
leg remuxes recording.mkv -> recording.mp4 (-c copy, no re-encode)
before the artifact upload, on all three OSes.
Verified end-to-end in a real browser against a generated artifact
zip: zip load, mp4 playback, timer, tab switching, follow-sync at
t=6/t=12, click-to-seek, autoplay policy (expected NotAllowedError on
synthetic play; real clicks fine).
Also fixes the shim fail message's dead variable ( ->
observed_git_url) in both posix drivers.
The app-update legs died on the onboarding overlay - a fullscreen div
that intercepts every click (the Settings click timed out under it).
The old plan seeded a fake provider key, which lies: the overlay
vanishes but the app is broken. Instead the driver now runs the
desktop E2E suite's own mock inference server
(tests-js/scripts/mock-server.ts, zero deps, bare-node type
stripping >=22.18) and configures it into HERMES_HOME byte-for-byte
like the dev:mock flow: config.yaml provider + MOCK_API_KEY env. The
app boots genuinely configured - no overlay, real chat surface.
e2e-assets/mock-provider.{sh,mjs} own start/stop (pid + url files;
the wrapper lives until SIGTERM - gating on stdin-close made the
server die instantly, a background process's stdin is already EOF)
and the config write. Wired into the posix script driver's
hermes-desktop-app-update arm and the macos driver's update phase
(both app-update methods). The Playwright flow keeps its defense-in-
depth: the real escape hatch ('I'll choose a provider later') and the
verified 'Open settings' selector.
Probed locally: models + streamed/non-streamed completions answer,
server stops cleanly on kill.
Playwright must own the spawn (it needs the inspection pipe), but
hermes desktop is not just build+launch - stamp checks, integrity
gates, sandbox fixups, and a constructed child environment. So the
driver intercepts the product's own launch: a sitecustomize.py on
PYTHONPATH (opt-in via HERMES_E2E_CAPTURE_LAUNCH) wraps subprocess.run,
captures argv/cwd/env at the spawn site, and fakes success instead of
spawning; launch-from-spec.mjs then _electron.launch-es exactly that
spec and clicks Settings -> About -> Update now. Completion is product
state, not a Playwright event: the handoff result file or the checkout
reaching the expected sha (source installs write no result file).
Ships with the driver, so it works unchanged on every sampled OLD ref
- no product flag, no pre-flag fallback split. Both launch shapes are
matched (npm exec electron / packaged exe under apps/desktop/release);
npm BUILD calls pass through untouched. Exit 0 without a capture fails
the leg: a version that never reached its launch must not pass.
Probe-the-probe: scripts/launch_capture_probe.sh runs control rows
(no opt-in, non-launch argv) and both treatment shapes - all green
locally. Gate flips on the shared run workflow for linux/macos;
windows adopts the same path with the driver restructuring.
The composite action .github/actions/e2e-screen-record owns setup and
lifecycle on all three OSes: ffmpeg via apt/brew-verify/winget+cache,
capture via x11grab/gdigrab/avfoundation, mkv at 15fps stopped by 'q'
on live stdin with kill fallback. Linux runners have no display, so
start brings up a dedicated Xvfb :99 and exports DISPLAY - one display
serves both the recorder and any app a later step launches.
Recording moves out of the GUI driver into workflow infrastructure -
that is what makes it uniform - and a missing ffmpeg or a zero-frame
file now FAILS the leg instead of skipping silently: the graceful-skip
path is how the windows leg shipped no recording.mkv while green.
Lifecycle proven locally: start against lavfi testsrc, q-stop, ffprobe
duration check (record-start.sh/record-stop.sh under nix ffmpeg).
Attempt 8 drove the ENTIRE GUI update click-path successfully: onboarding
dismissed, Settings opened, About opened, Update now clicked, updating
overlay shown. The hand-off log proves the real update then ran: desktop
(pid 8880) exited, venv unlocked, 'hermes update --yes --gateway --force
--branch main' fetched from serve.git, found 1 new commit, pulled, and
restored. Everything worked.
The only failure was the driver waiting on Playwright's app 'close'
event, which doesn't fire reliably when the Electron app self-quits for
the hand-off. Switch to the authoritative signal: poll for the
HERMES_HOME/.hermes-update-in-progress marker (or the result JSON, or a
genuine window-gone), which the hand-off writes ~4s after the click. The
PowerShell driver still owns asserting the OUTCOME (target sha, marker
cleanup, working hermes, relaunched app) after the driver returns.
Attempt 7 got the whole way into the GUI update leg: the installed
Electron app launched under Playwright, booted, composer attached, first
screenshot captured. It then couldn't find the settings gear -- the
ERROR screenshot showed why: a fresh install with no CONFIGURED provider
(the seeded .env key isn't read as model.provider) shows the onboarding
card ('Let''s get you setup with Hermes Agent'), which covers the shell
and its settings gear.
The update path needs no provider, so the driver now clicks 'I'll choose
a provider later' (with skip fallbacks) to dismiss onboarding and reach
the shell before looking for the gear. Harmless no-op when onboarding
isn't shown. Gear (aria-label 'Open settings') and About nav ('About')
selectors already match the real components.
Windows PowerShell 5.1 reads .ps1 without a BOM under the legacy OEM
codepage, mis-decoding UTF-8 bytes. Em-dashes/box-drawing survived in
comments through attempts 2-4, but the previous commit added an em-dash
INSIDE a double-quoted Write-Host string — the misdecode there ate the
quote boundary and cascaded into a whole-file parse failure at the Stage
step ('Unexpected token', 'string is missing the terminator').
scripts/install.ps1 documents this exact constraint ('pure ASCII for PS
5.1 parser compatibility'). Strip all non-ASCII from the .ps1 and .ahk
files (em-dash->--, arrows->->, box-drawing->-). drive-update.cjs keeps
UTF-8 (Node decodes it natively). Both PowerShell files parse clean.
Attempt 3's proof frames showed the install SUCCEEDED end-to-end
(bootstrap complete, installer self-copied to HERMES_HOME) and the
window advanced to 'HERMES IS READY' with a [ LAUNCH ] button at the
same centered CTA spot the [ INSTALL ] button occupied — screen (511,454)
inside window x=64 y=34 w=896 h=659.
Two Launch-step bugs, both fixed from that evidence:
* launch-button.png was the stale #68183 template and never matched the
restyled '[ LAUNCH ]' button. Re-captured from the live frame.
* the window-relative fallback used fy=0.59, clicking y=422 — above the
real button. Correct fraction is (454-34)/659 = 0.637. With the
template now matching, the fallback is belt-and-braces anyway.
Install click, completion detection, and the app-window wait were all
already correct in attempt 3; only the Launch click missed.
Attempt 2's proof frames showed two bugs, both now fixed from the live
evidence:
1. The #68183 install-button.png predated the installer UI restyle to the
'[ INSTALL ]' bracket look, so the template never matched and we fell
through to the position fallback. Re-captured install-button.png from a
real CI desktop frame (the actual rendered button).
2. The fallback then clicked the WRONG spot: ahk_exe's first WinGetPos
matched a hidden 16x16 helper window ('Window found at w=16 h=16' in
ahk.log), and BTN_FY=0.87 aimed below the real button anyway. The
button center measured at ~(0.50, 0.59) of the ~full-screen window.
Rewrite:
* WaitForRealWindow() skips phantom/hidden matches (requires w>400,h>300)
and returns the true rect; the installer window is then activated before
any click.
* Install-finished is now driven primarily by the authoritative
'bootstrap complete' line in bootstrap-installer.log (matches
BootstrapEvent::Complete), with the Launch template as a secondary
signal and a window-relative fallback click.
* Fallback clicks use the corrected (0.50, 0.59) window fraction.
Frame-0005 of the proof capture showed the exact failure: 'Unhandled
error: (6) The handle is invalid' rendered over the installer within
seconds of launch. AutoHotkey started via Start-Process has no console,
so FileAppend to '*' (stdout) throws — and the throw fired inside Log(),
killing the script before it clicked anything. The installer then sat
untouched at the INSTALL screen for 50 minutes.
* Log() now try-wraps the stdout write (file log is the real record)
* Install/Launch clicks fall back to the button's relative window
position when the #68183-era PNG templates don't match the restyled
UI ('[ INSTALL ]' bracket style visible in the same frame)
* install-finished has a second signal: 'bootstrap complete' in
bootstrap-installer.log (read with write-sharing), so a template miss
can't strand the wait
* driver passes the bootstrap log path as arg 3
Second job on the Windows E2E workflow covering the surfaces a user
actually touches, per Teknium's requirement:
* INSTALL: downloads the production Hermes-Setup.exe from
hermes-assets.nousresearch.com, launches it HEADED, and AutoHotkey
clicks Install -> waits -> clicks Launch (button templates + ImageSearch
approach from @ethernet8023's #68183, retargeted by process name and
extended to exercise the Launch hand-off). The real Electron Hermes.exe
window must appear.
* UPDATE x2: the installed Hermes.exe is launched under Playwright's
Electron driver and the test CLICKS Settings -> About -> Update now.
The production hand-off chain runs untouched: app quits, detached
updater (repo script or staged binary) runs hermes update, rebuilds
the desktop, relaunches Hermes.exe. Asserts: target sha, marker
cleanup, result JSON when the script path wrote one, working hermes,
and the RELAUNCHED app window. Leg 1 -> CURRENT, leg 2 -> synthetic
NEXT.
Proof artifacts: per-step renderer screenshots (booted app, settings,
About panel, update-available, updating overlay), full-desktop frames
every 3s across the whole run, ahk.log, bootstrap-installer.log,
desktop-update-handoff.log — uploaded on success AND failure.
The website exe runs exactly as shipped (its own pinned install.ps1,
its baked release-pin commit); the only environmental deltas are the
serve.git URL redirect, uploadpack.allowAnySHA1InWant for the commit
pin fetch, and a placeholder provider key so the update legs meet the
app shell instead of onboarding.
The contract job from the previous commits is unchanged and independent
— it remains the rollback position if the GUI job proves flaky.
Kuxin