Commit Graph

141 Commits

Author SHA1 Message Date
Teknium 53db597201 simplify(compat): hermes_state — drop 81 re-exports + 3 registry aliases + 3 shims, repoint 45 callers + 60 test files
hermes_state.py: delete every '# noqa: F401 (re-exported...)' import block (hermes_state_common/errors/guard/
readpool/sessions/fts/dbfile/wal/repair/registry + agent.context_compressor _DB_PERSISTED_MARKER_KEY); keep
only the names hermes_state.py itself uses, without noqa.
hermes_state_registry.py: drop get_shared_session_db/release_shared_session_db/close_shared_session_dbs
aliases; every caller (gateway/, tools/, tui_gateway/, cron/, mcp_serve, run_agent, tests) now imports
acquire/release/close_all/release_or_close from hermes_state_registry.
hermes_state_titles.py: drop set_auto_title_if_empty shim (title_generator keeps its getattr fallback).
Re-remove shim-only names restored by 34abf954bd: latest_user_message_row_id (tests call
latest_message_row_id(key, role='user'); role-targeting assertions kept) and get_session_activity (tests
build the snapshot via agent.session_activity.build_activity_snapshot over db.get_session(sid)).
hermes_state_wal._log_once resolves its dedupe sets as module globals instead of via hermes_state;
hermes_state_repair helpers call module globals directly (tests patch hermes_state_repair.<name>).
Frozen updater surface untouched (update_cmd_maint imports only SessionDB from hermes_state).
2026-09-03 13:46:50 -07:00
Teknium a9b0dd6742 simplify(compat): cron scheduler — drop 82 re-exports, repoint 8 callers + 33 test files
cron/scheduler.py no longer re-exports the split modules (scheduler_delivery /
_script / _prompt / _preflight); it imports only the 19 names it calls itself
(bottom-of-file, E402 kept for the import cycle). Dropped the shim-only
`import shutil` and the F401 note on windows_hide_flags (still used by
scheduler.py). Split modules now call same-module helpers directly, reach
sibling split modules via late-bound module refs (_delivery/_script/_preflight)
next to _sched, and import windows_hide_flags themselves; origin-resident names
(load_config, Path, _SCRIPT_TIMEOUT, heartbeat_run_claim, ...) still go through
_sched. Callers/tests import + patch the defining module.
2026-09-03 13:38:36 -07:00
Teknium f6938b37f3 simplify(compat): terminal/file/environments — drop 42 re-exports/aliases, repoint 20 callers + 41 test files
tools/terminal_tool.py: drop 30 pure re-export names (lifecycle/config/backends/
sudo/guards/result/interrupt/utils/_DockerEnvironment/is_managed_tool_gateway_ready)
and the noqa-F401 comments on the 25 names the facade itself uses. Sibling modules
(terminal_tool_backends/_result/_sudo/_lifecycle, environments/base, process_registry)
that read removed names through the facade now import from the defining module.
tools/environments/base.py: drop 11 re-exports (base_output/base_session_env/
path_utils) and the BaseEnvironment.stop() compat alias (no in-tree caller; the
lifecycle hasattr(env, 'stop') fallback stays for third-party envs).
tools/environments/docker.py: drop 1 re-export + the re-export comment.
Callers/tests repointed to tools.terminal_tool_{lifecycle,backends,sudo,config,
guards,result}, tools.interrupt, tools.environments.{base_output,base_session_env,
path_utils}.
2026-09-03 13:29:55 -07:00
kshitijk4poor db339f0051 fix(state): consolidate gateway SessionDB writers via process-wide shared registry
A gateway process opened state.db from ~12 call sites, each minting its
own writer connection, self._lock, close-time WAL checkpoint, and
token-writer thread. With N independent writers on one WAL file, one
connection's close-time checkpoint could race another's growth — the
lost/reordered-page-write signature across 11+ incidents (#90837).

Adds hermes_state_registry.py: a process-wide, per-path, refcounted
shared registry owning the writer boundary.

- acquire(path): same resolved path returns the same instance (one
  writer connection, one lock, one token-writer thread) for every
  long-lived in-process caller (gateway runner, SessionStore, per-agent
  lazy recall, cron per-job, mirror, channel_directory, slash_commands,
  shutdown_flush, session_search, react_to_message, delegate, mcp_serve,
  auto_archive, tui_gateway).
- close() on a shared instance is a NO-OP — the registry owns the
  lifecycle, so one caller's close can never tear down a writer other
  callers still hold.
- Generation-aware retirement on inode change: a replaced state.db
  RETIRES the live generation (never lent again) but keeps it alive for
  existing holders; release is object-keyed so holders of the old
  generation drain it independently of the new one. The old
  generation's own write path still fails with the typed
  StateDbReplacedError (existing protection, unchanged).
- Replacement-open failure leaves NO registry entry for the path —
  the next acquire retries fresh, never hands out a closed stale object.
- All teardown runs OUTSIDE the registry lock: a final release's WAL
  checkpoint can never stall acquisition for every state.db.
- close_shared_session_dbs() at gateway shutdown drains every
  generation (live + retired) as the final safety net.

CLI one-shots, recovery flows, and read-only cross-profile opens keep
using SessionDB() directly with their own close() — only long-lived
in-process sites route through the registry.

References #90837 (root-cause tracker stays open: the #10 EOF signature
and the WAL-lifecycle A/B verdict remain under investigation there).
2026-09-01 20:55:35 +05:30
Andrew Bagrin b7c59bda54 fix(cron): isolate per-execution working directories 2026-08-31 09:59:39 -07:00
Victor Kyriazakos 580daa7b96 fix(cron): mirror continuable-cron briefs for origin-fallback and opted-in explicit targets
A managed cron (created by a provisioning script, not from a live gateway
chat) never captures an origin. With cron.mirror_delivery: true and
deliver: origin, its brief was delivered to the home channel — the
user's own DM — but the transcript mirror and the in_channel session
seed were silently skipped: _target_matches_origin returns False for an
empty origin, and the whole continuable machinery keys off that check.
A user replying to the brief landed in a session with no record of it.
Field report 2026-08-17 (enterprise, Slack DM surface).

The June origin-scoping refactor (c06ceb3232) was written to exclude
broadcasts, and the exclusion is kept. What changes is the
classification: a home-channel FALLBACK for deliver=origin is the user's
primary conversation standing in for the origin, not a broadcast.

Changes:
- Delivery targets carry a resolution-provenance tag (_resolved_from:
  origin / origin_fallback / explicit; broadcast expansions untagged).
- _target_mirror_eligible replaces the bare origin check at the mirror
  gate: origin unchanged; origin_fallback eligible under the same flags
  as origin (per-job attach_to_session wins, else global
  cron.mirror_delivery); explicit platform:chat targets eligible ONLY
  under per-job attach_to_session — the global flag never activates
  them, so it cannot start writing transcript entries into arbitrary
  explicitly-addressed chats. 'all'/bare-platform stay never-eligible.
- Dedup OR-merges provenance so 'origin,all' resolving to the same chat
  keeps eligibility regardless of token order.
- _inchannel_seed_allowed guards the flat-session seed: group-channel
  session keys are user-isolated, so a seed without a user_id (origin-
  less job into a shared channel) would create an orphan session no
  reply resolves to — those targets fall back to the plain mirror. DM
  targets (keys don't embed user_id) always seed.
- cronjob tool schema text updated to describe the new attach scope.

Behavioral note: origin-less deliver=origin jobs under global
mirror_delivery now activate the full continuable path — on default
'thread' surface this opens a dedicated thread in the home channel
where the brief previously posted flat. That is the documented
continuable behavior; the silent flat post was the bug.

15 new tests (tests/cron/test_mirror_origin_fallback.py): eligibility
matrix (origin/fallback/explicit/all/bare/other-chat), dedup order
both ways, end-to-end mirror via _deliver_result for all four shapes,
origin regression control, seed user_id guard.
2026-08-26 16:06:09 +05:30
Teknium b96a9b7408 test(cron): delivery-targets test scopes platform assertions past bot-chat entries
cron_delivery_targets() now also lists machine-local bot-chat:<profile>
entries; the sibling test's exact set-equality assertion predates them.
Scope the platform assertions to gateway entries and pin that bot-chat
entries are always home_target_set.
2026-08-21 12:48:53 -07:00
Teknium ef04d846e9 feat(cron): cron agents now run with memory enabled like every other agent
Cron jobs were constructed with skip_memory=True and a hard 'memory'
toolset denial, so MEMORY.md/USER.md never loaded and the memory tool was
stripped even from per-job enabled_toolsets. That was inconsistent with
kanban/delegate/gateway agents (which all get memory) and forced users
into hacky bypasses.

- cron/scheduler.py: skip_memory=False on the cron AIAgent; drop 'memory'
  from _resolve_cron_disabled_toolsets; remove _strip_cron_memory_toolset
  and its call sites
- agent/agent_init.py: update stale comment referencing the cron denylist
- tests: flip pinning tests to the new contract (memory enabled, per-job
  memory toolset kept, user-level denylist still wins)
- docs: cron-internals + automate-with-cron no longer claim cron has no
  persistent memory
2026-08-21 03:46:37 -07:00
Adolanium fc9cbc872d fix(cron): do not load MEMORY.md into scheduled jobs
Cron already sets skip_memory=True and denylists the memory toolset.
The default cron toolset still names memory, so init treated that as a
request and built MemoryStore. MEMORY.md then landed in the job prompt.

Treat a denylisted toolset as not requested, and strip memory from the
cron enabled list. Flush agents that actually want the memory tool are
unchanged (#65429).
2026-08-21 13:24:43 +05:30
Ben Barclay cc85feeac7 test(cron): native scalar-fallback test asserts the live delivery actually ran
seed_mock.assert_not_called() alone could pass for the wrong reason — a
harness failure before delivery also leaves the seed uncalled. Assert
the real adapter recorded exactly one live send to the origin chat, so
the test pins the D6 thread-fallback decision, not an accidental
no-delivery.
2026-08-20 20:52:37 +10:00
Ben Barclay 540237c99e test(cron): pin the auto-mocked D6 accessor; prove the native scalar fallback with a real adapter shape
Unspecced MagicMock/AsyncMock adapters fabricate
supports_inchannel_continuable_for_platform as a truthy callable, so the
scheduler's duck-typed D6 gate silently took the relay accessor branch in
every in-channel test — the native scalar fallback the fixtures describe
was never exercised, and setting supports_inchannel_continuable=False on
a mock could not force thread mode. Pin the accessor to None on both
mock fixtures (matching a real native adapter, which never defines the
method), and add a fallback-boundary test with a real minimal adapter
class: scalar False -> in_channel fails safe to thread, flat seed never
fires.
2026-08-20 20:43:45 +10:00
Ben Barclay 4308c453ac fix(cron): stamp persisted origin scope_id onto origin-matching delivery metadata
The seed-key fix made the SESSION scoped, but the delivery leg still
dropped the scope: cron route_metadata carried only job_id (+thread),
DeliveryRouter stamps scope_id only for the configured HOME channel, and
the RelayAdapter's per-chat scope cache is cold after a gateway restart
(learned from inbound only). A scoped Slack origin that is not the home
chat therefore egressed with NO tenant discriminator, and the connector's
fail-closed guard could reject the brief before delivery — the
delivery-leg sibling of the seed-key scope gap.

Copy origin.scope_id into the live text and media routing metadata for
ORIGIN-MATCHING targets only (setdefault — never overrides router/home
stamping). Fan-out/broadcast targets are excluded by the origin gate: a
fan-out target's tenant is not the origin's, and a wrong scope is worse
than none.

Tests: restart-shaped positive (scoped non-home origin -> scope_id on
routed metadata, RED before this fix) and legacy negative (scope-less
origin stamps nothing).
2026-08-20 20:42:20 +10:00
Ben Barclay 20c56f82b8 fix(cron): in_channel thread-flatten uses the seed's gate (origin_target)
The seed was decoupled from the mirror opt-in (in_channel is the
continuation surface regardless of attach_to_session), but the
thread-id-clearing gate above it still read mirror_this_target. With the
advertised default config (attach_to_session=false, cron.mirror_delivery
unset) and an origin carrying a real thread_id, the brief kept delivering
INTO the origin thread while the flat (thread_id=None) session got
seeded — brief and continuation surface in different places, so a plain
reply never saw it.

Flatten on the same gate as the seed: origin_target (with the existing
live_adapter_ready guard). Fan-out/broadcast targets are unaffected.

Test drives _deliver_result with a thread-carrying origin and default
knobs, asserting on the routed DeliveryTarget.thread_id — RED on the old
gate, GREEN now.
2026-08-20 20:07:44 +10:00
Victor Kyriazakos 4673836f79 fix(cron): deterministic in_channel seed + companion thread-surface seed
Two live failures from the Alice canary (2026-08-19, jobs 28a24afebd81 /
83b93f8be379), both leaving a continuable in_channel cron with amnesia:

1. Seed mirrored via origin heuristics and silently dropped the brief.
   _seed_cron_channel_session created the flat session row, then
   mirror_to_session RE-DISCOVERED the target via find_session_by_origin —
   whose multi-candidate bail-out returns None on a populated chat (flat
   session + N per-message thread sessions sharing one chat_id, mixed
   user_ids). Receipt: 'in_channel seed did NOT land on slack:D0BJTDCSR7C'.
   mirror_to_session now accepts an explicit session_id and both cron seeds
   pass the exact row they just created; origin-scan remains the fallback
   for callers that genuinely don't know the target.

2. The brief's OWN THREAD was never seeded. in_channel delivers flat, but
   a flat Slack message still invites a thread reply (the natural mobile
   affordance — exactly what the user did). That reply keys to
   (chat, thread=<brief ts>), which no seed touched. The delivery's
   message_id now anchors a companion _seed_cron_thread_session so BOTH
   reply surfaces (plain channel message AND in-thread reply) continue the
   job.

Also: thread-seed failures upgraded debug→WARNING (silent seed failure IS
the user-facing bug), and the thread seed reports landed/not-landed.

Regression tests drive both against the live failure shapes: exact-session
mirror asserted via session_id kwarg; thread companion asserted via the
SendResult message_id anchor. Clean-fixture blind spot noted: the E2E
harness used a fresh store with one row, which is why heuristic rediscovery
looked fine pre-production.
2026-08-20 01:40:23 +00:00
Victor Kyriazakos 3c52d3589f fix(cron): in_channel seed must not require the attach_to_session mirror opt-in
Live regression (Alice, 2026-08-19): a continuable cron with
cron_continuable_surface=in_channel delivered its brief flat, but the
flat-session seed was gated on mirror_this_target = mirror_enabled AND
origin-match. Without attach_to_session (and with cron.mirror_delivery
defaulting False) the seed never ran; the next plain reply resolved to a
blank (slack, chat, None) session and the agent had no idea about its own
delivery message — not continuable in channel OR thread (in_channel mode
correctly skips thread creation, so there was no thread session either).

in_channel IS the continuation surface, not a mirror nicety: gate the seed
on origin-match alone, and resolve origin_user_id for any origin-matching
target so the seeded key still carries the scheduling user on
per-user-isolated chats. attach_to_session remains the opt-in for the
separate default-surface mirror behavior.

Regression test drives the delivery path with attach_to_session=False and
asserts the seed fires with the right user_id (fails on pre-fix code).
2026-08-20 01:40:23 +00:00
Teknium 07a5179158 Inspired by Poke: nudge review of repeatedly-failing recurring cron jobs
Poke (poke.com) 'encourages users to review recurring automations that
haven't been acted upon'. Hermes' equivalent pain point is a recurring
cron job that fails run after run: each failure delivers the same one-line
error with no signal that the automation itself needs attention.

- cron/jobs.py: persist a failure_streak counter in mark_job_run —
  incremented on agent failure, reset on success; delivery failures don't
  count. Back-compat: missing field reads as 0.
- cron/scheduler.py: _failure_streak_nudge() appends a review nudge to the
  delivered failure summary once a recurring job's streak reaches
  cron.failure_nudge_threshold (default 3, 0 disables). One-shots never
  nudge.
- hermes_cli/cron.py: 'hermes cron list' shows '(N failures in a row)' on
  failing jobs with streak >= 2.
- docs: new 'Repeated-failure review nudge' section in cron.md.

Tests: 17 passed (TestMarkJobRun + TestFailureStreakNudge); E2E verified
with real cron store in temp HERMES_HOME.
2026-08-16 22:08:59 -07:00
Andrew Fiebert c2a1179a34 fix(cron): fall back on transient DNS during provider resolve
Agent crons resolve OAuth credentials before the agent loop. A short
macOS/WARP DNS blip raised httpx.ConnectError ([Errno 8] nodename nor
servname provided) from xai-oauth token refresh, and the scheduler only
walked fallback_providers on AuthError — so Daily Focus Kickoff died
even when XAI_API_KEY / Anthropic were healthy.

Treat ConnectError/DNS OSError (and cause-chain equivalents) like
AuthError when selecting the fallback chain. Keep provider+model atomic.
Regression test covers the ConnectError path.
2026-08-14 21:55:14 -07:00
Evgenii acaafcc6bb fix(cron): make immediate execution race-safe
- claim_job_for_fire returns the atomically claimed snapshot with a unique
  fire owner; heartbeat_fire_claim renews the lease; mark_job_run fences
  terminal writes by expected_fire_owner so a stale worker cannot record
  over a replacement claim.
- run_one_job heartbeats the fire claim and forwards a combined cancel
  event (ownership loss OR external cancel) into run_job; the agent path
  is interrupted cooperatively and script-based jobs (no_agent + pre-run
  scripts) are hard-stopped with a process-tree kill (POSIX killpg
  SIGTERM then SIGKILL for surviving group members; Windows
  taskkill /T /F), with a bounded pipe drain so a SIGTERM-ignoring
  descendant cannot wedge the worker on communicate() EOF.
- Shutdown interruption is scoped to the exact execution token instead of
  the bare job ID, so a replacement run of the same job never consumes a
  stale interrupted flag.
- fire_claim_fence serializes save/deliver side effects per profile+job
  with a cross-process flock; remove_job prunes the fence-lock entry.
- Preserves upstream BaseException terminal recording (#73973),
  completed one-shot retention (#80624), blocked_config preflight
  (T1-26), and the advance_next_runs batch on top of current main.
2026-08-14 20:46:50 -07:00
Adolanium ed4f50de51 fix(send_message): hand unresolved cron and react targets to the adapter again
Restores pass-through behavior for cron delivery and react/unreact that
was lost when d409f6748 routed them through resolve_send_target. Stored
cron job targets the channel directory doesn't recognize (e.g.
telegram:ops-room on a fresh install, photon group GUIDs) used to go to
the adapter verbatim; after d409f6748 they were silently dropped. Same
for react on platform-native ids.

Adds an opt-in pass_unresolved_references flag to resolve_send_target,
passed only by cron and react. Model-facing send tool stays strict.
Plugin platforms with a parser stay strict for all callers. The optional
validator still has the final say over passed-through ids.

Follow-up fixes on salvage:
- Update test_cron_relay_delivery_guards.py mock lambdas to accept **kw
  (file added to main after PR branch point; lambdas didn't accept the
  new keyword argument)
- Consolidate duplicated pass-through blocks into _pass_through_unresolved
  local helper

Fixes #85128

Co-authored-by: Adolanium <Adolanium@users.noreply.github.com>
2026-08-15 07:07:40 +05:30
Teknium 541fad3c3c test(cron): reconcile summarizer tests with honest chain wording and composed no_agent gate
The cherry-picked tests predate #85508's honest fallback-chain phrasing
and each other: assertions pinned the old 'exhausted or unavailable'
literal and #83188's no_agent fallback-note behavior, which #77648's
mode gate supersedes (no provider classification at all for no_agent
jobs). Assert the composed contract instead.
2026-08-13 11:49:24 -07:00
cation98 efbbc993f6 fix(cron): avoid false provider failure summaries 2026-08-13 11:49:24 -07:00
liuhao1024 66c60f81b6 fix(cron): thread per-run prompt through cronjob(action='run') (#57331)
Salvaged from PR #57342 by @liuhao1024 (with the injection-scan half
from PR #57360 by @ghedeselmabot): cronjob(action='run', prompt=...)
silently discarded the prompt argument — per-run context never
reached the spawned cron session.

The prompt is now threaded as extra_prompt through the whole chain
(cronjob run action → _try_dispatch_background_run/_execute_job_now →
_run_claimed_job → run_one_job → run_job → _build_job_prompt) and
appended to the stored prompt under a '## Run Context' header for
that single fire only — never persisted to the job definition. It
passes the same strict _scan_cron_prompt injection scan as stored
prompts before firing, and works identically on the background and
sync fallback paths.

Test fakes across tests/cron/ updated to accept the new kwargs
(sibling-test blast radius from the signature change).

Co-authored-by: liuhao1024 <liuhao1024@users.noreply.github.com>
2026-08-06 23:14:55 -07:00
liuhao1024 fa96419993 test(cron): add _build_job_prompt extra_prompt regression tests
Pins the scheduler-boundary contract: extra_prompt is appended under
'## Run Context', does not mutate job['prompt'], and the header is
absent when extra_prompt is omitted.

Addresses review feedback from harjothkhara on PR #57342.
2026-08-06 23:14:55 -07:00
Paolo Shamoon 03dc4aad52 fix: hide memory tool from cron agents
Cron agents are constructed with skip_memory=True, so the memory
backend is not initialised — exposing the memory tool only gives the
model an unbacked tool that fails at runtime with 'Memory is not
available.'  Add 'memory' to _resolve_cron_disabled_toolsets() so the
tool is stripped from the schema before the model can call it.

Fixes #38129.

Co-authored-by: Paolo Shamoon <Paolo@Dylans-Mac-Studio.local>
2026-08-06 05:20:56 +05:30
Alex Fournier 884c2daa1c Merge updated tool metrics into skill metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	tests/tools/test_skills_hub.py
2026-08-02 20:12:37 -07:00
spfcraze 947310437b perf(cron): batch advance_next_run for the due-dispatch loop
The scheduler's pre-dispatch loop called advance_next_run per due job —
one full load_jobs() + one full save_jobs() of the jobs file each — so
N due jobs cost N reads + N writes of the whole file (gateway-restart
catch-up or co-scheduled bursts). advance_next_runs() does one load +
at most one save for the whole due set with identical per-job semantics;
advance_next_run() is now a thin wrapper over it.

Measured (50 due recurring jobs, real jobs file): 107.9 ms -> 2.5 ms
(45x; 50 loads + 50 saves -> 1 + 1).

Tests: batch advances recurring and skips one-shots, single load + save
I/O pin (fails pre-fix — no such function), no save when nothing
advances, and per-job wrapper semantics unchanged. Related: #60946 and
#75833 both restructure this loop's call site for correctness — neither
addresses the I/O cost, and this batch primitive composes with either
dispatch design; happy to rebase onto whichever lands first.
2026-08-02 21:15:16 +05:30
Alex Fournier 3d5fcab70f Merge updated tool metrics into skill metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	hermes_cli/observability/schemas/hermes.shared_metrics.v1.schema.json
#	scripts/smoke_nemo_relay_shared_metrics.py
#	tests/agent/test_skill_commands.py
#	tests/hermes_cli/test_relay_shared_metrics.py
#	tests/hermes_cli/test_relay_shared_metrics_runtime.py
#	tests/tools/test_skill_manager_tool.py
#	tests/tools/test_skill_usage.py
#	tests/tools/test_skills_tool.py
2026-07-31 07:30:30 -07:00
Teknium 6b81590c55 test: prune low-value tests suite-wide (wave 1) — 46,820 → 28,106 test functions
Systematic prune per AGENTS.md test policy, one pass over every major
test tree (gateway, hermes_cli, tools, agent, run_agent, plugins, cli,
cron, tui_gateway, honcho/openviking, root-level):

- DELETE: source-reading tests (read_text/getsource on prod files),
  change-detector tests (exact catalog counts, model-name snapshots,
  config version literals), mock-echo tests (assert a mock returns what
  it was told), assertion-free/trivial tests, near-duplicate
  parametrizations (boundaries + one representative kept), async/sync
  twin duplicates, cosmetic within-file variations.
- KEEP (mandatory): security/redaction/approval guards, message-role
  alternation invariants, prompt-caching/deterministic-call-id
  invariants, issue-number regression tests (deduped), E2E tests.
- 6 test files deleted outright (script-style/no-assert or fully
  redundant); conftest.py, fakes/, fixtures/ untouched.
- tests/acp/conftest.py added: autouse fixture stubs the live
  models.dev/GitHub/Copilot/Anthropic inventory fetches that ACP server
  tests performed on every session create — test_server.py 147s → 3.4s,
  and the tests are now genuinely hermetic.
- Sleep-based slowness shrunk where safe (codex_ttfb_watchdog,
  compression_concurrent_fork, etc.); no wall-clock assertion tightened.

Verification: full hermetic suite via scripts/run_tests.sh —
2439 files, 31,130 tests passed, 0 failed, 0 flaky retries, 315s wall
(baseline: 583s wall, 13,564s subprocess CPU).
2026-07-29 13:10:23 -07:00
Alex Fournier f1fd678e44 feat(observability): add Relay skill metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-29 12:20:34 -07:00
tachyon-r be05741349 test: isolate delegated and Photon environment state 2026-07-28 18:17:52 -07:00
kshitijk4poor 0aa15c08dd fix: update test stub to accept workdir kwarg from salvaged PR #70548 2026-07-24 15:55:39 -07:00
Victor Kyriazakos 45a408f41a fix(gateway): deliver relay-backed homes after restart 2026-07-24 10:45:13 -07:00
Jake Long Vu f279f7fcf1 Preserve Slack cron thread origin 2026-07-22 20:57:25 -07:00
Wesley Simplicio 0c07a19752 fix(cron): keep bare platform delivery on home target 2026-07-22 20:57:25 -07:00
Vinay Bikkina 141d5db922 fix(cron): scope in_channel thread_id clear to the live-send/seed gate
The in_channel flat-delivery clear was gated on `runtime_adapter is not
None`, but the flat continuation session is seeded ONLY on the live-send
path, which also requires a running event loop. When an adapter is
present but the loop is absent/not-running, the live-send/seed block is
skipped and delivery falls through to the standalone path — clearing
thread_id there flattened an unseeded brief with no continuable session
behind it (and bypassed the D6 capability check).

Factor the live-send condition into `live_adapter_ready` and gate both
the thread_id clear and the live-send block on it, so the clear can no
longer drift from the seed. Add an adapter-present/no-running-loop
regression test (verified it fails against the un-scoped clear).

Addresses review r3609147550.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-22 20:57:25 -07:00
Vinay Bikkina 7337a9d997 fix(cron): clear inherited thread_id for in_channel delivery
cron_continuable_surface=in_channel is meant to deliver a cron brief FLAT
into a channel (thread_id=None) so a plain channel reply continues the job
via the shared-channel session. The scheduler already seeds that flat
session (_seed_cron_channel_session) but never cleared the ORIGINAL
target/origin thread_id before routing the actual delivery, so a job
scheduled from inside a Slack thread still delivered into that origin
thread — the seeded continuable session then never matched where the brief
actually landed.

Clear thread_id once in_channel_surface is finalized, before it is re-read
for route_thread_id / the standalone fallback. Scope the clear to EXACTLY
the target that gets flat-seeded — the origin-continuable target
(mirror_this_target) on a live, in_channel-capable adapter
(runtime_adapter is not None):

- fan-out / broadcast / explicit-thread targets keep their thread_id — they
  are not continuable and are never seeded, so flattening them would
  reroute an explicit thread and could collapse two targets into duplicate
  flat sends;
- the standalone no-live-adapter path keeps the origin thread — it can
  never seed the flat session, and without an adapter the D6 capability
  fail-safe can't run, so flattening there would both drop the brief out of
  any continuable lane and bypass D6.

This keeps the clear in lockstep with the seed condition
(in_channel_surface and mirror_this_target) and the D6 fail-safe.
mirror_this_target / origin_user_id are computed earlier using the original
thread_id to match the origin conversation.

Regression tests in TestCronContinuableSurfaceInChannel:
- a job whose origin carries a thread_id must not forward that thread_id to
  the live adapter (DeliveryRouter folds target.thread_id into send
  metadata), and the seeded continuable session stays flat (thread_id=None);
- with no live adapter, the standalone send falls back to the origin thread
  rather than flattening.
2026-07-22 20:57:25 -07:00
Teknium 1f4eaec88a test(cron): assert no-rotation tip resolution is a finalize no-op
With compression.in_place defaulting True the cron session id never
rotates; get_compression_tip returns the input id. Pin that the
salvaged #67188 fix titles/ends the ORIGINAL cron session in that path
(and for falsy tip returns), i.e. zero behavior change when no
compression rotation happened. Also maps colingreig's contributor
email for attribution CI.
2026-07-22 16:58:47 -07:00
Colin Greig 7fa795a674 fix(cron): finalize compressed session tips (86e2darn2) 2026-07-22 16:58:47 -07:00
Trevor Gordon 9bf5822a2f fix(cron): robust session title generation (#50535, #50536, #50537) 2026-07-16 22:39:47 -07:00
Dan Schnurbusch 679487b807 fix(auth): enforce complete fallback routes
Skip provider-only setup fallbacks, keep fallback selection explicit for resumed sessions, preserve configured primary identity for cron drift checks, and make the auth lost-update regression deterministic.
2026-07-16 06:14:56 -07:00
Dan Schnurbusch f68fd80f41 fix(auth): preserve fallback routes and OAuth state
Switch provider and model together after setup-time auth failure. Serialize global auth-store merges under target-specific locks and preserve auth-to-shared lock ordering for profile OAuth refreshes.
2026-07-16 06:14:56 -07:00
kshitijk4poor ffc10cc659 fix(gateway): quiesce API and cron work during drains
Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
2026-07-13 23:26:10 +05:30
Teknium dabae386ef fix(cron): bind claim heartbeats to dispatch owner (#62155) 2026-07-10 19:12:35 -07:00
kshitijk4poor 713e50e7d2 fix: normalize against tools.skills_tool.SKILLS_DIR, the root skill_view enforces
The extracted normalize_skill_lookup_name() resolved trusted roots via
agent.skill_utils.get_skills_dir(), but skill_view() enforces
tools.skills_tool.SKILLS_DIR — a separate module attribute that callers
and 60+ existing tests patch directly. With the helper reading a
different symbol than the enforcer, any SKILLS_DIR patch (or future
divergence between the two resolvers) makes normalization disagree with
enforcement and absolute-path loads regress silently. Read SKILLS_DIR at
call time (deferred import, cycle-safe) with get_skills_dir() as the
fallback, and align the new tests to patch the enforced symbol.

Follow-up to the salvage of #59829 by @HexLab98.
2026-07-07 14:40:56 +05:30
HexLab98 e7082ea99f test(cron): cover absolute skill path normalization (#59824)
Add unit tests for normalize_skill_lookup_name and a cron scheduler
regression that absolute paths under the skills dir reach skill_view as
relative lookups.
2026-07-07 14:40:56 +05:30
kshitijk4poor 5986cdd380 fix(cron): deliver before tearing down the agent's async clients (#58720)
Defense-in-depth alongside the interpreter-shutdown guard: run_job closed
the cron agent's async resources (agent.close + cleanup_stale_async_clients)
in its finally block BEFORE run_one_job called _deliver_result, so a live
delivery could race a torn-down async client. run_job now accepts an optional
defer_agent_teardown holder; when set it hands the live agent back instead of
closing it, and run_one_job tears it down (via the extracted _teardown_cron_agent
helper) only AFTER delivery — in a finally so a failed run never leaks. Default
path (holder=None) is unchanged, so every existing caller keeps inline teardown.

Reorder approach based on #58777 by @LavyaTandel; reworked to keep a single
delivery site in run_one_job and add regression coverage.

Co-authored-by: LavyaTandel <lavya@loom.local>
2026-07-06 02:05:46 +05:30
kshitijk4poor fc31f14cda fix(cron): disambiguate Telegram forum vs channel DM topics at delivery time (#52060)
The #22773 heuristic classified any telegram:<positive_chat_id>:<numeric_thread_id>
cron target as a Bot API channel Direct-Messages topic and routed it via
direct_messages_topic_id, which nulls message_thread_id. A normal forum-style
topic inside a private chat has the identical shape, so every such cron
delivery landed in General instead of the target thread (#52060). It also
means the only way to address a genuine channel DM topic was this same
ambiguous guess.

Disambiguate with the real runtime signal instead: probe the live adapter's
get_chat_info once and route via direct_messages_topic_id only when the chat
is actually a channel; everything else (private forum topic, forum supergroup,
group) and any probe failure fall back to message_thread_id — parity with
0.16.0 and with live reply routing. This fixes forum-topic delivery AND makes
genuine channel DM-topic cron delivery work correctly.
2026-07-04 14:33:14 +05:30
Swissly 242c9639a8 fix(cron): prevent multi-target delivery loop crash on per-target failure
The standalone thread-pool fallback in _deliver_result() runs inside the
`except RuntimeError:` block (taken when asyncio.run() sees a running loop).
When future.result() raised there (SMTP ConnectionError, timeout, etc.), the
exception was NOT caught by the sibling `except Exception:` — it escaped
_deliver_result() and crashed the whole delivery loop, silently skipping every
remaining target. Multi-target delivery (e.g. deliver: 'email:a,email:b') is a
documented feature, so this broke a promised contract.

Wrap the fallback in its own try/except so a per-target failure is logged with
exc_info and the loop continues to the next target.

Fixes #47163
2026-07-01 03:48:37 -07:00
teknium1 ac3f4aed96 docs(cron): correct stale 'no new seed code' comments for in_channel
The in_channel surface DOES add a seed: _seed_cron_channel_session CREATES
the flat (platform, chat_id, None) session and mirrors the brief into it,
because mirror_to_session only APPENDS to an existing session and the flat
channel row is otherwise absent for a chat_postMessage delivery. Correct
the scheduler thread-skip comment and the test class docstring, which still
described the earlier 'let the existing mirror seed it' design.
2026-07-01 03:16:13 -07:00
Ben 2c84fb42b0 fix(cron/slack): CREATE the flat session for in_channel (mirror only appends)
Live testing exposed a real bug: an in_channel continuable cron delivered
flat to the channel (✅) but the reply did NOT continue the job — the bot
had no brief in context and confabulated the answer.

Root cause: mirror_to_session only APPENDS to a session that already
exists (_find_session_id → no-op when none matches); it never CREATEs one.
A flat (slack, chat_id, None) row is only created when a human posts a
top-level message the bot processes — a cron chat_postMessage delivery
never goes through the inbound handler, so the row is absent and the brief
is silently dropped. The prior impl relied on the bare mirror (F5/OQ-1
concluded "deletion only" — wrong).

Fix: _seed_cron_channel_session mirrors _seed_cron_thread_session —
get_or_create_session FIRST (chat_type = "dm" if is_dm else "group",
thread_id=None), keyed to the ORIGIN USER'S id, then mirror. The channel
session key embeds user_id (…:group:<chat>:<user>), so a system:cron id
would key the seed away from the reply; the origin user's id makes seed
key == inbound reply key. DM key ignores user_id but needs chat_type=dm
to match the prefix. Wired into the in_channel branch after delivery;
suppresses the generic mirror to avoid double-write.

DM validated (per request): the seeded key equals the inbound DM reply key
for a 1:1 DM; continuation works there too.

Tests:
- Rewrote the in_channel tests to use a real _session_store and the origin
  user_id; assert get_or_create_session is called with the flat, correctly-
  keyed source. Prove-fail: (a) reverting the create step and (b) seeding
  with system:cron each turn a targeted test RED; restore → GREEN.
- +2 direct _seed_cron_channel_session unit tests asserting the KEY-MATCH
  invariant (seed key == inbound reply key) via build_session_key, for both
  channel and DM.
- Rewrote tests/manual/cron_inchannel_e2e.py to drive a REAL SessionStore +
  real mirror_to_session + real _find_session_id + real build_session_key
  (no session-layer mocks — the old mocked E2E is exactly why the bug
  shipped). Asserts the brief lands in the transcript and the reply resolves
  to the same session, for BOTH channel and 1:1 DM.

Full relevant sweep: 283 passed.
2026-07-01 03:16:13 -07:00