3 Commits

Author SHA1 Message Date
Teknium 5820d0b0d5 refactor(gateway/config): extract the config.yaml phase to gateway/config_loader.py; table-driven from_dict; defensive collapse
- load_gateway_config (536 LOC) is now a thin orchestrator: legacy gateway.json
  -> config_loader.load_yaml_layer -> GatewayConfig.from_dict -> env overrides
  -> validation. The yaml phase lives in gateway/config_loader.py as small
  functions driven by tables: _TOPLEVEL_BRIDGE (23 top-level/nested
  gateway.<key> bridges with 5 fallback modes), _SHARED_KEYS (28 per-platform
  keys copied into extra, with per-platform restrictions and transforms) and
  _PORT_BRIDGE_KEYS. Logger name kept as "gateway.config".
- GatewayConfig.from_dict: shared pick()/key_label() helpers replace the
  repeated "top-level key present else nested gateway.<key>" blocks; warning
  order preserved.
- _normalize_unauthorized_dm_behavior / _normalize_notice_delivery unified into
  _normalize_choice; _ensure_platform_extra_dict -> _dict_slot (also used by
  persist_home_channel); _getenv_int removed (dead since the env pass moved to
  config_env, which has its own _int_or).
- Platform._missing_: one _add_pseudo_member helper for both branches.
- Single warning sites in _coerce_optional_positive_int and
  coerce_systemd_watchdog_seconds; _validate_gateway_config placeholder pass
  flattened; small to_dict/getter collapses. Ruff F401/SIM102 clean.
- tests/hermes_cli/test_config_read_guard.py: allowlist gateway/config_loader.py
  (same owner as gateway/config.py — the extracted load_gateway_config phase).

gateway/config.py 2684 -> 1319 LOC (-50.9%); largest function now
GatewayConfig.from_dict at 119 LOC. Resolved-config parity: 160 cells
(16 yaml fixtures x 10 env sets) byte-identical to the integration base,
including captured log records and stderr.
2026-09-02 16:48:36 -07:00
ethernet 969094e4d2 fix(tests): remove four shared-state and lifetime faults at high concurrency
The suite now runs as one job with high per-file concurrency. Four tests
depend on state that they share with their siblings, or on a timer that
outlives them. That was safe at 8 workers. It is not safe at 96 or more.
Runs 32547184159 and 32551746525 show them.

1. Every pytest subprocess shared one temp root.

pytest puts tmp_path under <temproot>/pytest-of-<user>/. At the end of a
session it walks that directory with cleanup_dead_symlinks(). The walk lists
the directory. Then it asks whether the `pytest-current` symlink resolves.
Then it unlinks the symlink. A second process replaces that symlink between
the question and the unlink. The first process then raises FileNotFoundError
after all of its tests passed. Two files failed this way and passed on retry.

scripts/run_tests_parallel.py now gives each subprocess its own temp root
through PYTEST_DEBUG_TEMPROOT, and deletes it after the attempt. No two
processes share a directory. The race has no shared object to act on.

Proof: a direct driver of _pytest.pathlib.cleanup_dead_symlinks against one
root, with a second thread that replaces the symlink, raises the same
FileNotFoundError on 'pytest-current' as CI. A private root for each
subprocess removes that condition. A separate check confirms that 5
subprocesses receive 5 distinct roots, that tmp_path lands inside the private
root, and that no root survives the attempt.

2. The config read guard walked directories that other tests were writing.

tests/hermes_cli/test_config_read_guard.py scanned the tree with rglob. rglob
descends into every directory and filters after that, so it calls scandir() on
__pycache__ trees that the guard never inspects. Sibling processes create and
delete those entries during the run. A directory that disappears in the middle
of a walk raises FileNotFoundError out of rglob.

The scan now uses os.walk. It prunes excluded directories before it descends,
and it ignores a directory that disappears. __pycache__ joins the excluded
set, because bytecode is not source.

The guard still catches what it exists to catch. With a planted raw
yaml.safe_load of config.yaml in hermes_cli/, the test fails and names the
planted file. With a clean tree it passes.

3. A PTY test waited for a file to exist, and not for its content.

tests/tools/test_process_registry_write_stdin_surrogates.py spawns a child
that runs open(out,'wb').write(sys.stdin.buffer.readline()). open() creates
the file empty. The bytes arrive only after the PTY delivers the line. The
wait stopped at out.exists(), which the empty file already satisfies, so the
read returned b'' when the parent won that gap. This test failed both attempts
in CI, and did not pass on retry.

The test now waits for the expected bytes, with a bounded deadline.

Proof: the old wait loses 6 times in 25 runs on an idle 16-core machine. The
new wait loses 0 times in 25.

4. A dialog close timer outlived the test that started it.

ConfirmDialog holds the "done" beat for 600ms after a successful confirm, then
calls onClose. The timer had no cleanup, so an unmount inside that window left
it armed. It then called onClose on a tree that is gone, which reaches
setState in the parent. vitest can tear the environment down first, and React
then reads `window` during the update:

    ReferenceError: window is not defined
     at resolveUpdatePriority (react-dom-client.development.js:1308)
     at dispatchSetState
     at Timeout.t4 [as _onTimeout] session-actions-menu.tsx:574

The frame at session-actions-menu.tsx:574 is the `onClose` prop of
DeleteSessionDialog. The owner of the timer is ConfirmDialog, which now keeps
the handle in a ref and clears it on unmount.

Zoomable had the same fault, with a 1500ms timer that clears a "copied" flag.
copy-button.tsx and tooltip.tsx already clear their timers.

Proof: a new test confirms, unmounts inside the 600ms window, then advances
the clock. Against the old code it fails with "expected onClose to not be
called at all, but actually been called 1 times". Against the new code it
passes.

Verification:
- The affected Python files and the tests of the runner itself pass under
  scripts/run_tests.sh.
- The desktop ui suite passes: 566 files, 5382 tests, and no
  "window is not defined".
- eslint reports 0 errors on apps/desktop. The 118 warnings are the state
  before this change. The two cleanup effects carry an eslint-disable line for
  the ref-mirror rule. They write a timer handle, and not a mirror of a
  reactive value. The rule permits this, and its own comment names the case.
- The PTY test cannot run on the NixOS development machine. That machine has
  no python3 outside the nix store, and the test uses the literal `python3`.
  The child exits 127 there. The fix rests on the 25-run measurement above and
  on CI.
2026-08-22 02:25:12 -04:00
teknium1 ed33ebca1d refactor: canonical config loaders for behavioral reads + guarded raw-read primitive (kills the managed-scope/env-expansion drift class)
The disease: ~15 scattered raw yaml.safe_load(config.yaml) reads that
silently miss managed-scope overlay, ${ENV_VAR} expansion, profile-aware
pathing, and root-model normalization. Every new config feature needed an
N-site sweep (incident chain 9cbcc0c9c8 → 732293cf87 → b0e47a98f9 →
1928aa0443). This commit assigns every raw read to an owner and adds a
lint-guard test so the class cannot regrow.

New primitive (additive-only change to hermes_cli/config.py):
  read_user_config_raw(path=None) — reads the user file EXACTLY as
  written; docstring states it is ONLY legal for write-back round-trips
  and raw-file diagnostics. Behavioral reads must use
  load_config()/load_config_readonly().

BEHAVIOR FIXES (class-a sites migrated to a canonical loader — these
previously read values that could DIFFER from the effective config):

  gateway/run.py _try_resolve_fallback_provider → _load_gateway_runtime_config
    keys: fallback_providers/fallback_model (provider, model, base_url,
    api_key). Drift fixed: a managed-pinned fallback chain was ignored;
    an api_key of "${OPENROUTER_API_KEY}" reached the resolver unexpanded.
  gateway/run.py GatewayRunner._load_provider_routing → same loader
    key: provider_routing. Drift fixed: managed-pinned routing prefs and
    ${VAR} templates were ignored.
  gateway/run.py GatewayRunner._load_fallback_model → same loader
    keys: fallback chain. Same drift as above.
  gateway/run.py GatewayRunner._refresh_fallback_model
    keeps the raw primitive (its last-known-good-on-parse-failure contract
    forbids the fail-open loader, which returns {} on a torn write) but now
    applies managed overlay + env expansion inline. Drift fixed: chain
    edits under managed scope / env templates were previously frozen out.
  tui_gateway/server.py _load_cfg (72 behavioral call sites)
    now = raw read + managed overlay (pre-existing) + NEW ${VAR} expansion,
    split from a new _load_cfg_raw() write-back primitive. Drift fixed:
    e.g. custom_prompt: "hello ${VAR}", agent.system_prompt, model,
    api_key/base_url templates reached sessions unexpanded. DEFAULT_CONFIG
    is deliberately NOT merged (callers treat missing keys as unset;
    `_load_cfg() == {}` sentinels and _save_cfg round-trips depend on it).
  tui_gateway/server.py _profile_configured_cwd
    keys: terminal.cwd of a NON-launch profile. Drift fixed: managed
    overlay + ${VAR} expansion now apply (load_config() would resolve the
    wrong profile's home, so the raw primitive + inline pipeline is used).
  plugins/platforms/telegram/adapter.py _reload_dm_topics_from_config
    → load_config_readonly(). keys: platforms.telegram.extra.dm_topics.
    Drift fixed: managed overlay + profile-aware pathing + expansion.
  plugins/memory/holographic _load_plugin_config → load_config_readonly().
    keys: plugins.hermes-memory-store.*. Same drift class.

WRITE-BACK ROUND-TRIPS (class-b: stay raw BY DESIGN via read_user_config_raw;
merging defaults/overlay would pollute the saved user file):
  gateway/slash_commands.py: model persist x2, _save_gateway_config_key,
    memory/skills write_approval toggles
  gateway/platforms/yuanbao.py auto-sethome
  tui_gateway/server.py _write_config_key + all cfg→_save_cfg blocks
    (reasoning show/hide/full/clamp, details_mode[.section], prompt)
    → new _load_cfg_raw()
  plugins/memory/holographic save_config

RAW-FILE DIAGNOSTICS + presence-sensitive bridges (class-c: stay raw,
now via the shared primitive with an explanatory comment):
  hermes_cli/doctor.py x5 (model validation, stale-root-keys, .env drift,
    deprecation sweep, memory-provider probe — the latter two keep their
    inline managed overlay where they had one)
  gateway/run.py _bridge_max_turns_from_config and the module-level
    TERMINAL_*/HERMES_* env bridge (bridging merged defaults would export
    all of DEFAULT_CONFIG into the environment; both keep their inline
    overlay + expansion)
  hermes_cli/send_cmd.py env bridge (same presence-sensitivity)
  hermes_cli/gateway.py multiplex-conflict probe (reads the DEFAULT root's
    config, not the active profile's — load_config is the wrong owner)
  hermes_cli/profiles.py / hermes_cli/web_server.py / tools/wake_word.py
    multi-profile reads (load_config targets only the ACTIVE profile home)
  cron/jobs.py _resolve_default_model_snapshot and cron/scheduler.py
    run_job config read keep their existing inline overlay+expansion but
    now share the primitive (their fail-open + last-value semantics and
    the deliberate no-defaults merge are preserved exactly).

Failure-semantics audit: every migrated site preserves its exact previous
behavior on missing file ({} / early return) and parse failure (raise into
the caller's existing except, warn, last-known-good, or fail-open) —
read_user_config_raw intentionally mirrors bare open()+safe_load semantics
(raises on parse errors, {} only on FileNotFoundError/non-dict root).

Guard: tests/hermes_cli/test_config_read_guard.py scans the tree for
yaml.safe_load within 6 lines of a 'config.yaml' reference outside an
explicit ALLOWLIST (hermes_cli/config.py, gateway/config.py, gateway/run.py
fallback path, hermes_cli/managed_scope.py which reads the MANAGED file,
gateway/readiness.py parse-health probe) and fails on new offenders.

E2E: tests/hermes_cli/test_config_loader_e2e.py runs a subprocess with a
temp HERMES_HOME (config.yaml containing ${E2E_PROMPT_SUFFIX}) plus a
HERMES_MANAGED_DIR overlay pinning agent.reasoning_effort, asserting
tui _load_cfg resolves "hello world"/"high" while _load_cfg_raw +
_save_cfg round-trip the template and user value verbatim with no
managed/default leakage.
2026-07-29 10:53:29 -07:00