ci: run the work lanes on larger runners and merge the split jobs
Every Linux lane that does real work ran on a 4-core `ubuntu-latest`. The Python suite and the JS checks were split into many small jobs to make that size usable. Each split job repeated the full setup. In most of the JS jobs the repeated setup cost more than the work. The work lanes move to larger runners. Then the splits that existed only to make small runners usable go away. Python tests: 12 slices become 1 job on a 96-core runner. Slicing cost a matrix job, a duration cache, a per-slice artifact and a merge job. 96 cores clear the floor that the slowest single test file sets, which is about 82s. A second slice divides work that is already at that floor, and adds a second setup. Duration data from run 32522943054 gives the numbers behind this: 3178 files, 11645s in series. The worker count is explicit, because `run_tests.sh` defaults to twice the core count. A later commit sets it from a measurement on this hardware. JS checks: 14 jobs become 1. The matrix paid about 371s of repeated setup to spread about 612s of work. One larger runner installs one time. The three UI shard scripts and `run-ui-shard.mjs` are therefore removed, because the unsharded `test:ui` covers the same tests. The unit of parallel work inside that job is a CHECK, and not a workspace. apps/desktop is most of the payload, and its own `check` is a serial && chain. A spread across workspaces alone therefore leaves that chain as the long pole. A package that declares `check:*` sub-scripts gives one unit for each sub-script. That is the same selection rule the matrix used. The loop lives in `.github/scripts/run-workspace-checks.mjs`, so the same sequence runs on a laptop. It runs 11 units together, buffers the output of each one, and fails at the end with the full list. Children that share one stdout interleave their lines and make a failure hard to read. `npm run --ws check` stops at the first workspace that fails. `check:test:plugins` joins the desktop `check` script. The matrix prefers `check:*` sub-scripts over the plain `check` script, so `check:test:plugins` ran only as its own leg. Without this change the merge drops that suite and the job stays green. node_modules is cached on the lockfile, and `npm ci` is skipped on an exact hit. The `cache: npm` option of `setup-node` caches only the ~/.npm tarball cache, which leaves the extract and the postinstalls to pay again. The arm64 image build stays on a native arm64 runner. A build of linux/arm64 on an x64 host uses emulation. The docker test lane caps its workers at the core count. Each of those tests drives a container, so the docker daemon sets the limit and not the processor. `.github/actionlint.yaml` declares the runner labels. actionlint knows the GitHub-hosted labels only, and an undeclared label reads as an error that hides the real findings. The `detect` job checks out one file through a sparse checkout, and its timeout drops to 1 minute. It reads `scripts/ci/classify_changes.py` and nothing else. Verification: - actionlint reports 9 findings across all workflows. An unmodified HEAD with the same config reports the same 9. This change adds none. - A wrong label still fails. actionlint reports `ubuntu-latest-32-cor` and `ubuntu-latest-32-arm-cores`. - Every changed workflow parses, and `name` parses as a string. - A replay of the `save-durations` merge step against a three-artifact layout returns all 3178 entries. - An expansion of the npm script graph gives the same leaf commands for the parallel units and for a plain `npm run check`, in both directions. Against the 13-leg matrix the count is 13 to 11, and the whole difference is the three UI shards that collapse into one unsharded `check:test:ui`. - `--list` reports the 11 units, and a full local run completes and reports the time of each unit. - The runner labels cannot be verified here. The first real run is the test.
This commit is contained in:
ethernet
+20
-93
@@ -2,11 +2,6 @@ name: Tests
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
slice_count:
|
||||
description: Number of parallel test slices
|
||||
type: number
|
||||
default: 8
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -17,42 +12,17 @@ concurrency:
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
generate:
|
||||
name: "Generate slices"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
outputs:
|
||||
matrix: ${{ steps.matrix.outputs.matrix }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
|
||||
- name: Restore duration cache
|
||||
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
with:
|
||||
path: test_durations.json
|
||||
key: test-durations
|
||||
# Saves use test-durations-${run_id}, so the exact key above never
|
||||
# matches — without this prefix fallback the cache ALWAYS missed,
|
||||
# LPT slicing ran on no data, and unbalanced slices pushed heavy
|
||||
# files toward the per-file timeout under load.
|
||||
restore-keys: |
|
||||
test-durations-
|
||||
|
||||
- name: Generate test slices
|
||||
id: matrix
|
||||
run: |
|
||||
MATRIX=$(python3 scripts/run_tests_parallel.py --generate-slices ${{ inputs.slice_count }})
|
||||
echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT"
|
||||
|
||||
test:
|
||||
name: Run tests slice ${{ matrix.slice.index }}/${{ inputs.slice_count }}
|
||||
needs: generate
|
||||
runs-on: ubuntu-latest
|
||||
name: Run tests
|
||||
# One 96-core runner for the whole suite. There is no slicing. Slicing
|
||||
# existed to spread the suite over 4-core runners. It cost a matrix job, a
|
||||
# duration cache, a per-slice artifact and a merge job to do it.
|
||||
#
|
||||
# 96 cores clear the floor that the slowest single test file sets (about
|
||||
# 82s). A second slice divides work that is already at that floor, and
|
||||
# adds a second setup.
|
||||
runs-on: ubuntu-latest-96-core
|
||||
timeout-minutes: 30
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJSON(needs.generate.outputs.matrix) }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
@@ -117,73 +87,30 @@ jobs:
|
||||
# re-download, keeping the persisted cache small and fast to restore.
|
||||
run: uv cache prune --ci
|
||||
|
||||
- name: Run tests (slice ${{ matrix.slice.index }}/${{ inputs.slice_count }})
|
||||
- name: Run tests
|
||||
# Per-file isolation via scripts/run_tests.sh: each test file runs
|
||||
# in its own freshly-spawned `python -m pytest <file>` subprocess
|
||||
# with bounded parallelism. No xdist, no shared workers, no
|
||||
# module-level state leakage between files.
|
||||
#
|
||||
# File list is pre-computed by the generate job (--generate-slices)
|
||||
# which runs LPT distribution once and passes the file list to each
|
||||
# matrix job via --files. Previously each job re-discovered files and
|
||||
# re-ran LPT independently — redundant N times.
|
||||
# No --files: the runner discovers the suite itself. The discovered
|
||||
# set is identical to the list the removed matrix job used to pass in.
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
scripts/run_tests.sh --files '${{ matrix.slice.files }}'
|
||||
scripts/run_tests.sh
|
||||
env:
|
||||
# This is the maximum number of test FILES that run together.
|
||||
# run_tests_parallel.py starts one pytest subprocess for each file
|
||||
# from a single ThreadPoolExecutor, so this value IS the limit. The
|
||||
# default is cpu_count*2, which is 192 here.
|
||||
#
|
||||
# A later commit sets this from a sweep on the real runner.
|
||||
HERMES_TEST_WORKERS: 144
|
||||
# Ensure tests don't accidentally call real APIs
|
||||
OPENROUTER_API_KEY: ""
|
||||
OPENAI_API_KEY: ""
|
||||
NOUS_API_KEY: ""
|
||||
|
||||
- name: Upload per-slice durations
|
||||
# Advisory artifact (feeds slice balancing) — a transient artifact-
|
||||
# service blip must not fail an otherwise-green test slice.
|
||||
continue-on-error: true
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: test-durations-slice-${{ matrix.slice.index }}
|
||||
path: test_durations.json
|
||||
retention-days: 1
|
||||
|
||||
# Merge per-slice duration data into a single cache, so future runs
|
||||
# (including PRs) get balanced slicing.
|
||||
save-durations:
|
||||
needs: test
|
||||
if: needs.test.result == 'success' && github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Download all slice durations
|
||||
# Each slice uploads the same file name (test_durations.json).
|
||||
# With merge-multiple, the parallel downloads write to one path.
|
||||
# This causes two problems: a race can write two JSON documents
|
||||
# into one file, and the last write erases the other slices.
|
||||
# Without merge-multiple, each artifact gets its own directory.
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
pattern: test-durations-slice-*
|
||||
path: durations
|
||||
|
||||
- name: Merge into single durations file
|
||||
run: |
|
||||
python3 -c "
|
||||
import json, glob, os
|
||||
merged = {}
|
||||
for f in glob.glob('durations/*/test_durations.json'):
|
||||
with open(f) as fh:
|
||||
merged.update(json.load(fh))
|
||||
with open('test_durations.json', 'w') as fh:
|
||||
json.dump(merged, fh, indent=2, sort_keys=True)
|
||||
print(f'Merged {len(merged)} file durations')
|
||||
"
|
||||
|
||||
- name: Save merged duration cache
|
||||
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
with:
|
||||
path: test_durations.json
|
||||
key: test-durations-${{ github.run_id }}
|
||||
|
||||
e2e:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
Reference in New Issue
Block a user