`submit` measures the scroll jump when a turn is appended; nothing measured
the jump when a session is opened, which is the prepend/settle path. Clicks
sidebar rows and tracks how far the bottom turn moves after first paint.
Adds an `idle-cost` scenario for the symptom Brooklyn reported: with a
thread spinning, resizing the sidebar feels slow. It holds N tiles busy,
pushes NO tokens, and measures the renderer's self-inflicted commit rate
plus fps while dragging the splitter and while typing.
It reproduces immediately. Five busy tiles, nothing streaming:
idle commits 17.7/sec (should be 0 — nothing is happening)
drag 1.4 fps p95 812ms, worst frame 1.9s
typing 61 fps (fine — this is specific to resize)
Attributing the drag window showed 105,385 TooltipProvider renders and
~15s of component time across a 60-frame gesture. Cause: `Tip` mounts a
full Radix provider + Tooltip per call site, and there are ~107 of them.
Radix's Tooltip holds real state and Popper subscribes to layout, so an
unrelated interaction re-rendered all of them.
Mounts the machinery lazily instead, on first hover/focus. Tooltip churn
drops ~4x (105k -> 26k) and drag doubles to 3fps.
Note `defaultOpen` on the armed Tooltip is load-bearing: the pointerenter
that armed it has already fired, so Radix never sees it and the tip mounts
silently closed. A test caught exactly that, and now guards it.
3fps is still bad — the remaining cost is the whole transcript
re-rendering per resize frame (MessagePrimitive.Parts 12,600 renders /
10.5s, Block/Ct 24,300 each, all 100% wasted). Separate fix.
Drives the same synthetic multi-tab streaming workload as `multitab`
(publishSessionState per session per flush, no backend, no credits) but
reports render attribution instead of frame pacing — sidebar_renders,
wasted_renders, and wasted_notifies.
Answers 'does the sidebar re-render while an agent is typing' directly.
The existing `stream` scenario measures streaming into an empty thread, which
is exactly the case where per-delta transcript work does not show up. This adds
`--historyTurns`, which mounts a settled transcript and lets it drain BEFORE
the recorders start, so the measurement window contains streaming work only —
and `stream-history`, a report-only preset that turns it on.
Report-only (tier: manual) rather than gated: how much history a host can mount
varies, so the absolute number is not comparable across machines. It is meant
for same-machine before/after runs.
Co-authored-by: Jakub Wolniewicz <frizikk@users.noreply.github.com>
N session tiles stacked as tabs, all mounted (keep-alive) and all
streaming concurrently through the real publishSessionState path — the
"several PR reviews at once" workload. Frame pacing + longtask metrics,
no backend or credits needed; the workload that exposed both fixes above
and the regression gate that keeps them fixed.
Stop drip-feeding scenarios: extend the harness to cover the latencies that
actually dominate perceived speed, and measure them on a REAL production build.
- --prod: build a production renderer with the probe included (VITE_PERF_PROBE=1,
off in normal builds) and launch it from dist/. Measures minified React, so
numbers are representative shipped figures instead of ~3x-inflated dev ones.
- cold-start scenario (tier "cold"): launch → CDP → driver → first paint, via a
fresh isolated spawn per run. Captures spawn_to_cdp_ms, spawn_to_driver_ms, fcp_ms.
- first-token scenario (backend tier): Enter → first assistant token painted —
the TTFT latency an agent app is uniquely judged on.
- run.mjs gained --prod (build once), cold-start fresh-spawn loop, and gates
ci+cold tiers against the baseline.
Baseline re-captured on a PRODUCTION build (median of 5), darwin-arm64 — all
green. Representative numbers:
cold-start spawn→interactive ~1.6s, FCP ~0.5s
stream frame p95 22ms, 1 longtask
keystroke p50 2ms, p95 8.7ms
transcript mount 145ms, 82ms longtask (400-msg open)
The prod build also settled the open question from the dev numbers: the
transcript-mount "lead" (221ms longtask in dev) is only ~72-82ms in prod — not
actionable. Measurement did its job.
Replaces the dozen ad-hoc measure-*/profile-* scripts (each reinventing the
CDP client — 4 different copies — plus its own arg parsing, stats, output
path, and none with a baseline) with one framework under scripts/perf/:
- lib/cdp.mjs one CDP client + target discovery + typing + CPU-profile wrapper + DOM selectors
- lib/stats.mjs percentiles, histograms, CPU-profile self-time ranking
- lib/baseline.mjs load/compare/update baseline + regression gate (new capability)
- lib/launch.mjs attach, OR spawn a fully ISOLATED instance
- scenarios/* one module per measurement, registered in scenarios/index.mjs
- run.mjs / serve.mjs, baseline.json, README.md
Isolation solves the long-standing measurement blocker: a running `hgui` held
the Electron single-instance lock, so a second instance quit. `--spawn` /
`perf:serve` launch with their own --user-data-dir (separate lock scope), their
own HERMES_HOME (separate backend/sessions, config seeded from ~/.hermes so it
reaches a chat view without onboarding), and their own --remote-debugging-port.
Synthetic scenarios drive $messages via window.__PERF_DRIVE__, so no LLM credits.
Scenario -> sunset script mapping:
stream <- measure-synthetic-stream, profile-synth-stream, profile-long-stream
stream --real <- measure-real-stream, profile-real-stream
keystroke <- measure-latency, profile-typing, leak-typing
transcript <- (new: long-transcript mount cost)
submit <- measure-submit, measure-jump
session-switch <- profile-session-switch
profile-switch <- measure-profile-switch
CPU profiling is now a cross-cutting --cpuprofile flag, not 5 separate scripts.
CI-tier scenarios (stream, keystroke, transcript) need no backend/credits and
are gated against baseline.json (seed values; re-capture with --update-baseline
on a reference device). Backend-tier scenarios are report-only.
perf-probe.tsx gains loadTranscript() for the transcript scenario. No core
files touched; isolation is via CLI args, not env-gated app changes.
Verified: node --check all modules, tsc, eslint, and a unit smoke of the
stats + regression-gate logic. The end-to-end GUI run (which opens a window)
is left to run interactively via `npm run perf -- --spawn`.
Johnny