scripts/run_tests.sh tests/gateway/ on a clean macOS checkout fails 7 tests
that pass on the Linux CI lane. The macOS CI lane runs only -m macos_only,
so these had never executed on a Mac. Each failure was diagnosed
individually, not assumed to share a cause:
* test_systemd_notify::test_notify_supports_systemd_abstract_socket
binds a NUL-prefixed abstract AF_UNIX name, a Linux kernel feature.
-> @pytest.mark.linux_only (the abstract socket IS the subject).
* test_shutdown_forensics::test_spawns_subprocess_and_writes_output
spawn_async_diagnostic() wraps a Linux-only script (ps --sort,
/proc/loadavg, dmesg, pstree) in GNU `timeout`, which macOS lacks.
Popen raises, the producer returns None by design (fail-soft).
-> @pytest.mark.linux_only, replacing a bare skipif(win32).
* test_scale_to_zero::test_suspend_self_{posts_suspend,non_2xx}
bound the fake flaps socket under pytest tmp_path, which exceeds
sun_path (104 B on macOS/BSD, 108 on Linux). Production connects to the
fixed /.fly/api so the code is not length-exposed; only the fake needed
a short home. -> short_sock_dir fixture (gettempdir -> /tmp fallback,
skip if neither fits). Verified with a 130-char TMPDIR on Linux.
* test_buzz_adapter::test_live_media_redacts_long_path_before_bounding
built 160-char path components; a single component is capped at 255 B
on both hosts and the full path hit PATH_MAX (1024) on macOS. The
invariant (path redacted BEFORE the 900-char bound) needs the path to
exceed 900 chars, not 1000+. -> grow 80-char components until the path
passes _MAX_CLI_MESSAGE_CHARS, assert < 1024. Mutation-checked: moving
the redaction after the bound fails the test.
* test_wecom_callback::{test_build_event_extracts_text_message,
test_poll_loop_dispatches_handle_message}
NOT environmental: `_build_event` parses with defusedxml, which ships
only via the `wecom` extra (or transitively via `youtube` -> [all] in
CI). A dev venv without it gets ET=None. -> dependency skipif on
DEFUSEDXML_AVAILABLE (not an OS marker; the 4 other tests in the file
still run).
Verified: scripts/run_tests.sh on macOS: 226 passed, 4 skipped (2
linux_only, 2 defusedxml). python:3.11-slim container with [dev,wecom]:
-m "linux_only and not integration" selects and passes both marked tests;
all 5 files 229 passed.
* feat(gateway): let NAS broker the scale-to-zero suspend where the guest has no lever
* fix(gateway): require the going_idle ack and outlast the broker before suspending
* fix(gateway): release the redial hold on every path that abandons the suspend
* fix(gateway): hold the re-dial only for the brokered lever, and require it
* fix(gateway): make the re-dial hold contract real and fence the in-guest suspend too
* refactor(gateway): share the watcher-iteration and descriptor helpers across the sleep tests
* refactor(gateway): move the sleep tests' repeated setup into their fixtures
* refactor(gateway): one docstring line per sleep test, and parametrise the abort and lever paths
* fix(gateway): fence the freeze gap, cool down aborts, and stop clobbering a shutdown drain
* fix(gateway): slice the Fly freeze fence on the wall clock and widen it for larger machines
With the gateway owning the suspend, the idle predicate covers every
work source (agent turns, cron jobs, API-server runs, background work,
fail-awake on unreadable sources) and the relay drains + flips before
the freeze, so a long timeout no longer buys safety — real work always
blocks the suspend and resume is sub-second. 5 idle minutes just bills
idle RAM. Per-instance override stays config.yaml
gateway.scale_to_zero.idle_timeout_minutes (D2).
New behavior-contract test: invalid config values degrade to the module
default (whatever it is), never zero/negative — asserts the RELATION,
not the literal, per the no-change-detector-tests rule.
Address sol-reviewer findings:
- The shared shutdown-drain counters swallow exceptions to 0 — fine for
a drain, unsafe for a suspend predicate (a transient read failure made
live work look idle, reopening the mid-job freeze). The suspend path
now reads both sources itself and treats an unreadable source as work
(sentinel 1, fail-awake) with a debug log. A MISSING api_server
adapter remains a normal not-work state.
- is_idle()'s parameter renamed running_agent_count -> active_work_count:
it receives the broad aggregate, and the old name invited future
callers to pass only agents again.
- New failure-path tests: unreadable cron source and unreadable API
source each hold the machine awake (both fail against the fail-open
shape); missing adapter stays idle-capable.
Fly Proxy autostop judges idle exclusively on inbound proxied connections.
It cannot see an in-flight agent turn (outbound-only LLM traffic), and since
Fly's mid-2026 proxy change an open outbound socket (the relay WS) no longer
holds a machine awake. With autostop:"suspend", Fly suspended machines while
they were still processing long-running jobs, and could suspend before the
gateway flipped the relay destination (the buffered-event black hole).
The scale-to-zero watcher now owns the suspend: after the idle predicate
holds (no running agents, no live background work, inbound-quiet) and the
go_dormant() quiesce completes (relay drained + flipped), it POSTs
/v1/apps/{app}/machines/{id}/suspend on the local /.fly/api flaps socket.
Suspend is skipped when the quiesce fails or inbound lands mid-quiesce
(flip-before-freeze), and off-Fly the step is a no-op (fail-awake).
Pairs with the NAS change that provisions scale-to-zero machines with
autostop:"off" (gateway-owned suspend); wake is unchanged (Fly-proxied
wakeUrl poke + autostart).
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
The gateway-side BEHAVIOUR layer that consumes the relay scale-to-zero
primitives (gateway-gateway Phase 5): the gateway decides it is idle and
drives the relay transport dormant so the platform (Fly autostop:"suspend")
can suspend the now-traffic-idle machine, which wakes on the connector's
wakeUrl poke (decisions.md Q3=C', D1-D13).
- gateway/scale_to_zero.py: pure helpers — scale_to_zero_enabled (the NAS
Labs HERMES_SCALE_TO_ZERO stamp, D11/Q8=A), parse_idle_timeout_seconds
(config.yaml gateway.scale_to_zero.idle_timeout_minutes, D2),
messaging_is_relay_only_or_absent (F6/D1), should_arm (D1/D11/§3.4(1)),
is_idle (D2/D3/F7).
- gateway/run.py: _last_inbound_at clock stamped on user inbound in
_handle_message (F13); the arm-gate + idle predicate + the
_scale_to_zero_watcher dormant sequence (mark draining -> adapter
go_dormant() -> cooldown), started only when armed. Deliberately NOT the
stop path and NOT mark_resume_pending (F12/D13).
- tools/process_registry.py: has_any_active() for the bg-work guard (D3/F7).
- hermes_cli/config.py: gateway.scale_to_zero.idle_timeout_minutes default 5.
Tests: 38 pure-logic + 6 watcher (incl. bg-work regression guard proven RED).
Full relay + scale-to-zero suites: 184 passed. The 20 unrelated failures in
the broader run are PRE-EXISTING on origin/main (custom-provider/tools tests),
confirmed via a pristine baseline worktree.
Ben Barclay