Commit Graph

4 Commits

Author SHA1 Message Date
kshitijk4poor 905fdda100 fix(conformance): make the journal-mode matrix honest — steer the child resolver, audit the effective mode
Salvage round on the Phase 1 skeleton (three findings from review):

1. The DELETE leg was silently upgraded to WAL on healthy SQLite:
   pre-seeding the file via PRAGMA was undone by SessionDB.__init__'s
   apply_wal_with_fallback(), which upgrades any non-WAL file whenever
   the configured mode (default wal) says so — only WAL-reset-vulnerable
   interpreters preserved DELETE, i.e. the leg tested the advertised
   mode only where CI wasn't running. Each matrix leg now pins
   database.journal_mode in an isolated HERMES_HOME for the child and
   audits the ON-DISK mode after the run (effective_mode_or_skip):
   a leg that ran in a different mode skips instead of double-counting.

2. Cell 2 exit-code conflation: a claimant crashing with an unhandled
   exception exits 1 — indistinguishable from the clean "lost the
   claim" exit(1), so one winner + seven crashes passed as consume-once
   proof. Codes are now disjoint (0=won, 10=lost, anything else=crash).

3. Crashed-writer diagnostics: wait_for now fails immediately with the
   child's stderr when the writer dies before reaching the kill window
   (was: 60s opaque deadline, stderr discarded). spawn_child prepends
   to an inherited PYTHONPATH instead of clobbering it.

Plus: dead `if False` scaffolding removed from cell 1's writer; README
matrix section corrected (cell 2 is default-mode-only by design — the
consume-once property rests on a single predicated UPDATE).
2026-08-24 15:54:23 +05:30
Ayush Nangia b392591d7a test(conformance): crash/resume persistence cells — Phase 1 skeleton (#80921)
Deterministic, LLM-free conformance cells against the real SessionDB with
real SIGKILL mid-write, per the tracking issue's spot-probe method:

- cell 1: acknowledged-append durability + recovery determinism (adapted
  from the issue's 29.5K probe, scaled kill window, identical assertions)
- cell 2: consume-once under 8-process concurrent claim_handoff
- cell 3 (new): compression-rotation atomicity — never a compression-ended
  parent without a continuation (#80337 contract; #80487 recovery context)
- cells 4-5: documented stubs interlocked with #82956-#82959 and
  #83197/#83557

Journal-mode matrix (resolver default / DELETE / WAL-with-skip-gate) per
cell; every wait deadline-bounded; writers asserted alive at kill time.
2026-08-24 15:37:26 +05:30
Teknium 39975613b1 test: prune wave 2 + speed fixes — 28,106 → 19,757 test functions, suite wall 315s → 294s
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.

Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
  (DEFAULT_CONFIG smart-approval leaked in) — pinned approval
  mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
  silently probed live provider catalogs (~2s/test) — stubbed
  cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
  shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
  MagicMock agents — interrupt.side_effect now clears _running_agents
  (22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
  (24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
  patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
  5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
  with rationale comment — the watchdog-dump assertion needs the loop
  blocked well past deadline+grace under parallel load (flaked once in
  the 40-worker verification run at a 0.2s margin).

Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
2026-07-29 13:39:40 -07:00
Ben Barclay 29fc746350 feat(conformance): vector generator — native renderers as executable spec (#71666)
* feat(conformance): vector generator — native renderers as executable spec

- scripts/generate_conformance_vectors.py: renders a 44-case corpus
  (markdown grid + scar tissue + adversarial agent output) through the
  NATIVE renderers (Telegram MarkdownV2 format_message, Slack mrkdwn,
  WhatsApp, Discord) and emits per-platform JSON vectors stamped with the
  oracle commit. Expect semantics: parity | semantic | divergent(note).
- tests/conformance/: committed vectors + 7 behavior-contract tests
  (determinism, self-free oracle invocation, shape, scar-bug coverage,
  committed-vectors-reproduce lockstep — the openapi.json discipline).
- Consumed by gateway-gateway's conformance runner (committed vectors +
  sender-level vitest suite + weekly refresh workflow).

* fix(conformance): explicit utf-8 encoding on vector file I/O (Windows footguns gate)
2026-07-26 16:52:25 +10:00