hermes-agent/evals/botmode-dm-delivery

Native Bot Mode delivery probe

Real Electron, production Python backend and tool execution, disposable HOME/HERMES_HOME, loopback scripted inference (no paid model). Linux seat fixture; run from repository root:

npm ci --no-audit --no-fund
cp evals/botmode-dm-delivery/probe-dm-delivery.spec.ts apps/desktop/e2e/
git apply evals/botmode-dm-delivery/mock-trigger.patch
(cd apps/desktop && npm run build)
(cd apps/desktop && DISPLAY=:0 XAUTHORITY=/run/user/1000/xauth_cnpsqU \
  XDG_RUNTIME_DIR=/run/user/1000 VIRTUAL_ENV="$VIRTUAL_ENV" \
  HERMES_DESKTOP_CDP_PORT=off npx playwright test e2e/probe-dm-delivery.spec.ts --reporter=list)
git apply -R evals/botmode-dm-delivery/mock-trigger.patch
rm apps/desktop/e2e/probe-dm-delivery.spec.ts

Use the current seat's actual Xauthority path and an existing runtime venv. Artifacts default to /tmp/botmode-dm-review/native (override with BOT_DM_EVIDENCE); sandbox path is printed. The fixture's generated hermes shim pins every child to this checkout, not an installed launcher.

Verified results

Base cf35e7351e770: nested beta quiet CLI executes real message_agent to the named alpha Desktop owner. Admission, execution and reply succeed, and the quiet sender receives its completion. A reload then shows the attributed incoming message. The live pre-reload view showed the reply but omitted the incoming row in this fixture; that renderer-refresh behavior is not fixed by this cron change.

Base CLI-owner case: separate cron producer returns exact SESSION_NOT_OWNED refusal. Release owner, run scheduler tick, open Beta in Desktop: no cron output, assertion red. Fixed case: queued receipt; release owner; real synchronous scheduler tick drains; Beta Desktop renders CLI_OWNER_CRON_SENTINEL and its reply once. Final two-case native run: 2 passed (1.2m). Supported-owner nested case also passed independently on base (1 passed (40.0s)).

This does not establish native Windows parity, general retry after unowned CLI failure, or correction of the issue #105460 --in ~ premise. CLI title resolution is profile-DB-based, not workspace selection; the named live-owner route is positive.

The queue is deliberately at-most-once after claim. A crash before spawning but after claiming remains inspectable as claimed; it is not retried automatically.

Independent review follow-up

The probe now admits the named Beta destination from the default scheduler, then changes the ticker's HOME to a different existing directory before drain. Published head c91dfcbfe810c fails with Profile 'beta' does not exist, leaving zero sentinel inputs in Beta. The follow-up pins the admitted home and ID; Beta renders one input and reply. Set BOT_DM_CORRUPT=1 to add one malformed JSON record alongside the valid admission: before the follow-up the real tick raises JSONDecodeError; afterward the damaged record stays on disk while Beta delivers.

Fresh built native run with both root change and corruption: 2 passed (1.4m), including the existing nested message_agent control. Receipts/screenshots: /tmp/botmode-dm-review/{native-red2,corrupt-red,final-native} and matching .log files. The first follow-up run also exposed a fixture mistake (the changed HOME was not created, so --in ~ correctly refused); that failed receipt is retained in native-green.log, and both source legs were rerun with an existing HOME. Prior /tmp/botmode-dm-recovery* evidence remains untouched.

Per-record delivery exception isolation

Set BOT_DM_EXCEPTION=1 and select -g "cron output" for the controlled native exception probe. exception-tick.py raises PermissionError at the actual post-discovery target Path.is_dir() boundary, not at the delivery helper. The same exception was first reproduced with real directory traversal permission loss on Python 3.11/Linux; the retained test uses a portable controlled fault. Before the guard, native tick exits 1, leaving the head claimed and sibling queued. Afterward, the head is ambiguous, the sibling settles and renders once, and a second real tick replays neither. Logs: /tmp/botmode-dm-exception-{red,green}.log; receipts and screenshot: /tmp/botmode-dm-exception/{red,green}/.

The review's repeated-head starvation claim is not reachable: the claim commits before delivery and later scans skip every non-queued record. One failed tick is real; recurring replay of that same head is not. Indefinite queued/payload retention is intentional, with no TTL or automatic ambiguous retry introduced here.

Ordinary custom-root fallback (#104066 / #104055)

probe-cron-root.spec.ts adds the never-deferred sibling: copy it to apps/desktop/e2e/ and run with the same native fixture (no mock-trigger patch needed for this case). It keeps default's real Desktop Bot Chat lease, submits ordinary cron output to unowned Alpha from a separate Python producer under a custom Hermes root, and holds the real quiet CLI child at loopback inference. The child shim PID must match Alpha's real CLI lease; default's lease is unchanged. After release, Alpha has exactly one input and Desktop renders the output. The same case removes unused Beta and verifies delivery neither recreates Beta nor creates a second .hermes root under HOME.

Both origin/main's scheduler and pre-follow-up c827ae179d67c fail with Profile 'alpha' does not exist before any recipient turn. Fixed native run: 1 passed (46.3s). Two invariant tests exercise the actual CLI startup resolver across named/default/own destinations with a changed active profile, and refusal when the destination is missing initially or disappears during discovery: 5 failed before, 5 passed after. The old env-clearing test is replaced by these behavior checks rather than retaining the broken expectation.

Evidence: /tmp/botmode-cron-root/{before2,origin-main,after}.log, after/{owners.json,children.log,rows.json,result.json,missing.json,ordinary-recipient.png}. The first fixture attempt (before.log) used the wrong default row label; the actual Desktop label is Hermes. No production failure is claimed for that attempt. Full cron directory: 1346 passed, 1 skipped across 116 files; sibling mailbox, DM, gateway consumer and profile tests: 124 passed, 3 skipped across 4 files. Credit @fangliquanflq's #104066 for the root-boundary diagnosis and anchoring fix; this combined branch reuses its already-resolved destination instead of repeating name resolution. No retry or receipt semantics change.