Byte-identical bodies moved out of hermes_cli/kanban_db.py into four sibling
modules, re-exported from the origin so kanban_db.<name> keeps resolving and
stays the single monkeypatch target; origin-resident helpers are reached via a
late-bound _kb namespace. AST-identity verified for all 329 moved symbols; SQL
statement multiset parity vs base. Three source-inspection tests repointed at
kanban_db_dispatch; the _add_column_if_missing alias test now imports the real
owner (hermes_cli.sqlite_util).
hermes_cli/kanban_db.py 12,153 -> 11,157 LOC, behavior-neutral.
- _row_get / _json_or / _json_dict / _env_int / _git_out applied at every
duplicate site; Event/Comment/Attachment.from_row row mapping.
- Scalar-read helpers (_task_status, _latest_event, _scratch_workspace,
_has_spawnable, _current_run_id, _parents_satisfied) replace repeated
conn.execute(...).fetchone() blocks.
- _migrate_add_optional_columns and add_notify_sub column refreshes are
table-driven (same statements, same order).
- _dispatch_lane_task unifies the ready/review lane tails; _claim_and_open_run
and _reclaim_dangling_run shared by claim paths; block_task's three UPDATE
branches and complete_task's UPDATEs collapsed to one statement each.
- connect() PRAGMA/WAL setup shared via _open_configured; _probe_integrity
shared by the health guard and repair; _try_lock_nb/_unlock shared by the
init and dispatch file locks; _insert_comment for in-transaction audit rows.
- Dead: parent_results (zero refs).
- Long docstrings hand-compacted, every invariant kept. Schema SQL untouched.
purge_stale_done_notify_subs only matched status='done', so a task the
circuit breaker parked in 'blocked' kept its notify-sub rows forever on
boards that never archive. Widen the predicate to done OR blocked while
keeping the existing age clause; backlog/ready cards are idle, not
abandoned, and stay exempt (test_gc_spares_reopened_task_even_when_old).
Watcher comment/log and docs updated to say done/blocked.
Closes#100955
Co-authored-by: itsflownium <itsflownium@users.noreply.github.com>
_sqlite_connect opened a connection via connect_tracked and then ran the
busy_timeout PRAGMA; if that raised, the half-open connection was abandoned
— leaking its fd AND leaving a stale entry in the sqlite_safe_read
live-connection registry (which only clears on close), permanently blocking
byte-level probes of the kanban database. Close before re-raising.
Salvaged from PR #96290 (kanban slice) with regression test.
Addresses three gaps found in review of the memory-guard PR:
P1a — standalone daemon was the one uncapped entry point. run_daemon()
now resolves kanban.max_in_progress every tick (explicit config wins,
else the memory-derived default) exactly like the gateway dispatcher and
`hermes kanban dispatch`. New shared parser configured_max_in_progress()
so all three entry points agree on what "explicitly configured" means.
P1b — max_in_progress was enforced per board while the gateway ticks
every active board, multiplying the host budget by the number of boards
(2 boards x cap 2 = 4 workers on a host sized for 2). The cap is now
host-level: _dispatch_once_locked() adds count_running_tasks_other_boards()
to the running count before deriving the tick's spawn budget. Enforced in
the shared locked path, so gateway, CLI, and daemon all inherit it.
max_spawn deliberately keeps its historical per-board semantics.
Fails open per board so one corrupt board can't brick dispatch on the rest.
P2 — the ready loop consumed the entire shared spawn budget before the
review loop ran, so a sustained ready backlog starved autonomous reviews
indefinitely. When spawnable review work exists (assigned + real profile,
mirroring the review loop's own gate) and the tick has budget, one slot
is held back from the ready lane. Reservation is per-tick and
self-releasing; the review lane still spends from the shared budget —
it gains fairness, not extra capacity.
11 new tests in tests/hermes_cli/test_kanban_host_cap.py. Existing
kanban suites: 278 passed (15 failures pre-existing, identical on clean
main baseline). ruff clean.
Two production incidents (OOF-77 "larrikin-lollies", OOF-30
"synclare-task-manager") followed the same shape: no
kanban.max_in_progress configured, a busy board, and a 1 GiB hosted VM.
The dispatcher fanned out 26-31 concurrent workers, the host went into
swap-thrash/OOM, and the whole machine — dashboard included — became
unreachable. NAS restart loops then masked the problem: each restart
"recovered" briefly before the kanban dispatcher immediately respawned
unbounded workers.
Building on the cherry-picked max_in_progress-across-both-lanes fix
(PR #28695, credit @Dusk1e), this adds two complementary safeguards to
hermes_cli/kanban_db.py:
1. Memory-DERIVED default concurrency cap. When kanban.max_in_progress
is unset, resolve_max_in_progress() derives a default of
clamp(MemTotal / 512 MiB, 2, 8) — e.g. 2 workers on a 1 GiB VM,
8 on 4 GiB+. Explicit config always wins in either direction. On
hosts where total memory can't be read (macOS/Windows dev machines),
the default stays None (no cap — unchanged behaviour). Wired into
both dispatch entry points (gateway/kanban_watchers.py and
hermes kanban dispatch) so behaviour matches regardless of path.
2. Live memory-PRESSURE guard inside dispatch_once. A static cap can't
see the host's actual memory state (other tenants, bloated
long-lived workers). The dispatcher now samples system memory each
tick via gateway.lifecycle_ledger.sample_memory() and classifies it
with gateway.memory_status.classify_pressure() (same thresholds as
the dashboard memory banner and OOM-suspicion heuristics from
NS-608/NS-656): critical -> spawn nothing this tick; elevated ->
at most one new worker; unknown -> no restriction (fail-open).
Reclaim/promotion bookkeeping still runs under pressure, and
deferred tasks stay queued — nothing is dropped. Restriction is
surfaced on DispatchResult.memory_pressure and logged.
Tests: tests/hermes_cli/test_kanban_memory_guard.py (14 tests) covers
the derived cap (floor/ceiling/fail-open/explicit-config-wins), the
pressure classifier, and dispatch behaviour under critical/elevated/
unknown pressure including defer-not-drop and bookkeeping-still-runs.
An autouse fixture in tests/conftest.py pins the memory sample to
"no data" suite-wide so existing dispatch tests don't depend on the
CI runner's live memory state (opt-out marker: real_memory_guard).
Kanban worktree workspaces were never removed by anything: _cleanup_workspace
preserved them by design, the CLI startup pruner explicitly defers t_* trees
to 'hermes kanban gc', and gc only sweeps scratch — so every worktree task
leaked its checkout forever (measured ~130GB on one estate).
- _cleanup_workspace now dispatches worktree workspaces to a new
_cleanup_worktree_workspace, which removes the worktree and its
auto-generated wt/<task-id> branch only when the tree is clean AND every
commit is reachable from a remote-tracking ref (reusing cli.py's
_worktree_is_dirty / _worktree_has_unpushed_commits predicates). Any
doubt preserves the worktree. dir workspaces stay untouched.
- The #33774 active-children deferral now covers worktree parents, and
_try_cleanup_parent_workspaces reaps deferred worktree parents when the
last child reaches a terminal state.
- archive_task reaps workspaces too; tasks archived without completing
previously leaked forever.
- 'hermes kanban gc' gains a backstop sweep for archived worktree tasks
that predate these hooks.
Tests: tests/hermes_cli/test_kanban_worktree_teardown.py (10 cases: removal,
dirty/unpushed/custom-branch/main-checkout/non-git preservation, complete/
archive integration, deferred-parent handoff).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`connect()` caches every path it has initialized in the process-local
`_INITIALIZED_PATHS` set and then skips all first-open work for it —
header validation, integrity probe, `SCHEMA_SQL`, additive migrations.
That cache is keyed on a path, but the schema it stands for lives in a
file, and the two can drift apart: delete or replace `kanban.db` under a
live gateway/dispatcher/dashboard process and the next `connect()` takes
the fast path, lets SQLite create a fresh empty database, and hands back
a connection with no tables in it.
Nothing notices. Every query then fails with `no such table: tasks`,
`plugin_api._conn()` logs its init warning and carries on, and the board
renders empty. Because the cache entry survives, the process re-creates
the same schema-less ~4 KB file on every restart of the desktop app in
front of it — only killing the backing process clears it.
Verify the sentinel table on the fast path and self-heal when it is gone:
drop the stale cache entry and fall through to the existing init path,
which re-runs the probes and the schema script under the cross-process
init lock. The check is one `sqlite_master` lookup on the already-resident
page 1, so the steady-state path stays lock-free (#36644) and does no
schema work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Now that subscriptions survive `done` (completion is reversible —
on every 5s notifier tick forever. Add
kanban_db.purge_stale_done_notify_subs(): one DELETE removing subs
whose task has been done with no new events past a retention window
(age = latest task event, falling back to completed_at/created_at, so
any activity exempts the task; a reopened task is exempt by status
alone). The notifier watcher runs it per board once at startup and at
most hourly, re-reading kanban.done_sub_retention_days (config.yaml,
default 30; 0 disables) at each sweep.
_inherit_notify_subs (link_tasks / triage-decompose / create-parents path)
copied only platform/chat/thread/user/profile, dropping chat_type,
user_id_alt, delivery_mode, and delivery_metadata. A DM-originated child
completion then fell back to chat_type='group' and woke a fresh
group-scoped session instead of the originating DM; Telegram DM-topic subs
lost their persisted reply-fallback metadata (issue #73030).
Consolidates the duplicated inline inheritance block in create_task onto
the single-owner helper — one inheritance path, every column, ONE owner.
Sabotage-verified regression tests for both the link_tasks and
create-with-parents paths.
api_server is stateless — its adapter has no push send(), so the wake
self-post IS the delivery on that path. Defaulting those subscriptions to
plain 'notify' left them with no delivery mechanism at all (the notifier's
doomed send() failed 12 times then dropped the sub), regressing the
pre-delivery_mode behavior and failing
test_apiserver_sub_wakes_real_session_via_self_post in CI slice 5.
Explicit modes still win; other platforms keep the 'notify' default.
Before delivery_mode existed the notifier woke unconditionally when the task
carried a session_id — pre-existing gateway subscriptions had de facto active
wake. The column's 'notify' default alone would silently disable that on
upgrade. Backfill gateway rows to notify+wake on first-add only (tui stays
notify); explicit user downgrades are never overwritten by re-migration.
Sabotage-verified regression tests included.
Salvage of #37865 by @verybigdog. Adds delivery_mode (notify / notify+wake / wake)
on kanban notify subscriptions, persists chat_type + user_id_alt so a woken turn
reconstructs the creator's real session key, inherits the return path to child
tasks, and keeps wake out of the model-exposed send_message schema.
Original commits were authored under a local placeholder identity
(hermes-agent@users.noreply.local); re-attributed to the contributor's
public email.
Implements the remaining observers from RFC #58548 (@thebizfixer),
accepted as the design basis in the #64231 batch disposition:
- on_kanban_worker_spawned: fires in the dispatch loop after spawn_fn
returns and the worker PID is durably persisted (the RFC timing
contract), in both the ready and review lanes.
- on_kanban_worker_exited: tick-derived from detect_crashed_workers;
fires after every reclaim/accounting txn has committed, carrying
exit_kind / exit_code / outcome / retry_status.
- on_kanban_worker_stale_claim: fires when release_stale_claims
reclaims a TTL-expired claim; live-PID claim extensions and deferred
reclaims stay silent.
- on_kanban_task_updated: task-mutation boundary observer carrying
changed_fields (field names only); fired by assign_task,
set_model_override, and set_reasoning_effort, and by the dashboard
plugin API's direct-SQL priority/title/body editors (single and
bulk) through the new kanban_db.notify_task_updated seam.
- on_kanban_dispatch_tick: re-port of PR #56066 (@laboratoiresonore),
renamed per the taxonomy and fired strictly AFTER _dispatch_tick_lock
is released; the sweeper found the original fired inside the lock,
where a slow subscriber could extend the single-writer critical
section and stall a sibling dispatcher.
All five are observer-only (return values ignored), fire after the
relevant write txn commits, and short-circuit on has_hook() so nothing
is built when no consumer registers; every fire site is fully
best-effort so a broken plugin can never break dispatch or a task
mutation. No config surface added. Existing plugins and hook payloads
are untouched.
Mutation-boundary scope: every user-facing task-FIELD editor fires
(assignee, priority, title, body, model/provider override, reasoning
effort). Deliberately not wired: status transitions (they belong to
the lifecycle hook family), dispatcher bookkeeping columns
(worker_pid, workspace_path, claim columns — surfaced through the
worker hooks instead), link/comment/attachment tables (not task-row
writes), and the dispatcher's default-assignee auto-assign (already
surfaced via DispatchResult.auto_assigned_default in the tick
payload). notify_task_updated is the seam for wiring further paths.
Docs: new rows plus a detail section in the shipped plugin-hook catalog.
Tests: 30 new (9 worker lifecycle, 8 dispatch tick, 8 task updated,
5 dashboard mutation boundary), including a lock-probe contract test
that fails if the tick hook ever fires inside the dispatch lock.
Refs: RFC #58548, #64231 batch disposition, folds #56066.
Follow-up to the state.db PASSIVE checkpoint salvage (PR #84277,
#45383/#80255/#44795): the kanban dispatcher's periodic explicit
checkpoint still used TRUNCATE on the shared kanban.db. The dispatch
flock only serializes dispatchers — CLI kanban commands in other
processes write to the same board without it, so the TRUNCATE races
live writers exactly like the state.db close() path did.
Switch it to PASSIVE and bound the -wal file with
journal_size_limit=8MiB set at connection init (SQLite trims the file
on the writer's natural post-checkpoint reset), since PASSIVE never
truncates.
tests/hermes_cli/test_kanban_db_repair.py updated to assert PASSIVE
and reject TRUNCATE. Remaining TRUNCATE call sites are test fixtures
operating on private temp DBs (sole opener), which is the legitimate
use.
Ancestor-reopen descendant invalidation previously lived only in the
dashboard plugin (_set_status_direct), so board semantics diverged by
surface and the retraction was silent: completed work snapped back to
todo and live workers were killed with no operator-visible signal.
Move it into kanban_db.invalidate_descendants_for_parent_reopen as THE
single domain implementation (recursive-CTE discovery and per-run
_retry_status_for_run handling preserved). It composes under a caller's
open transaction via write_txn(allow_nested=True) — the ancestor flip
and the descendant retractions must commit atomically — and opens its
own transaction standalone. The dashboard shim now delegates; the CLI
deliberately has no done-reopen verb (reopen-review is review-phase
only), so the DB-layer function being the single implementation is the
fix, documented in its docstring.
Non-silent: every invalidated descendant gets a descendant_invalidated
event ({ancestor, prior_status, new_status, resume_status}), the legacy
status event for existing live-feed consumers, and a task comment
naming the reopened ancestor. Running descendants keep the termination
behavior (a child building on a retracted premise is wasted spend), but
the events/comment are committed BEFORE the kill, which routes through
_terminate_reclaimed_worker — the same helper the reclaim paths use.
consecutive_failures resets to 0 on invalidated descendants: operator-
initiated invalidation is a deliberate fresh start, deliberately the
opposite of the review-loop rule (reopen_review_task preserves the
counter, #35072) so the autonomous review loop can't launder its own
failure streak.
Regression: DB-function reopen demotes done descendants with events +
comments; running descendant's audit trail is durable before its worker
dies; counter resets; dashboard and DB paths produce identical task
states, event kinds, and comment counts.
request_changes and reopen_review_task no longer reset
consecutive_failures (and last_failure_error) to 0 — review transitions
are neither success nor failure signals, so the circuit-breaker counter
is preserved (not incremented either), mirroring unblock_task (#35072).
Only complete_task's success path clears the counter.
Regression: counter=1 survives a full request_review -> request_changes
-> re-request cycle; a crash after request_changes accumulates to 2 and
trips a failure_limit=2 breaker; complete_task still resets to 0.
request_review on a running task under a live claim now requires the
caller to prove ownership (expected_run_id, the unchanged worker path)
or pass an explicit force=True override (CLI --force; dashboard human
actions pass force=True) instead of silently clearing claim_lock /
worker_pid of a live run.
Failures now carry distinct diagnostic reasons via with_reason=True
(mirroring request_changes' tuple pattern): live-claim refusal,
malformed re-review provenance, unsatisfied parents, unknown task, and
CAS miss. Tool/CLI handlers surface the specific reason instead of the
generic 'unknown id or not in running/ready'.
Regression tests: live-claim refusal + force/worker paths; malformed
provenance gets a distinct reason and explicit reviewer= recovers.
Thread lane= into check_respawn_guard. For review-lane dispatch the
active_pr and recent_success rules are skipped: a fresh PR URL comment
(and often a recent completed run) is the precondition of the canonical
review handoff, not a duplicate-work signal. Rate-limit cooldown and
the auth-blocker check still apply in every lane.
Regression: a review task with a <24h PR comment is spawned by dispatch
while a ready-lane task with the same comment stays deferred; a
rate_limited latest run still defers the review lane.
Plain write_txn raises loudly on nesting again (the historical main
invariant); composition primitives (create_task, add_comment) opt in
with allow_nested=True for savepoint semantics. create_swarm activates
the swarm root with an inline blocked->done CAS flip + synthesized run
+ event instead of nesting complete_task, so complete_task's post-commit
side effects (workspace cleanup, failure-counter clear, recompute_ready)
can no longer fire under an open outer transaction; recompute_ready now
runs after the outer commit. recompute_ready docstring corrected.
Regression: plain nesting raises; allow_nested composes and an outer
rollback discards inner work with no side effects fired.
Add a non-terminal "review" status so a worker that finished implementation
can hand off for human review without abusing kanban_block. The old
kanban_block(reason="review-required: ...") convention routed the handoff
through the unblock-loop breaker, so a normal review -> changes -> review
cycle was falsely escalated to triage.
- kanban_db: request_review (running/ready -> review, non-block, emits
review_requested), reopen_review_task (review -> ready/todo, review_reopened),
complete_task accepts review -> done, and a review_dispatch gate (default off,
shared by the dispatcher loop and the gateway health probe).
- kanban_request_review worker tool + `request-review` / `reopen-review` CLI
verbs; tool wired through toolsets, EXPOSED_TOOLS, _POLISHED_TOOLS.
- Gateway notifier wakes the origin subscriber on review_requested and
block_loop_detected; the subscription survives until done/archived, so every
review cycle re-notifies.
- Dashboard PATCH + bulk route the review transitions (request_review /
reopen_review_task) and render the review column.
- goals.py goal-loop and KANBAN_GUIDANCE recognize review as a terminator.
- Docs (reference tables, user guide, AGENTS.md, zh-Hans mirrors) + tests.
needs_input / failed are unchanged: they still route through kanban_block,
still count toward block_recurrences, and still escalate to triage.
Two small config-gated features:
1. Kanban orphaned-card reconciliation (kanban.reconcile_orphans, default
true, config.yaml): a running card with broken claim bookkeeping
(claim_lock or claim_expires NULL — crash mid-claim, manual SQL, DB
restore) is invisible to all existing recovery paths
(release_stale_claims requires claim_expires NOT NULL,
detect_crashed_workers requires host-local lock + pid,
detect_stale_running is config-disabled by default) and shows Running
forever. New reconcile_orphaned_running() pass in kanban_db.py runs
each dispatch_once tick: requeues orphans to ready with an explanatory
comment, closes any leaked run, emits a 'reconciled' event, and defers
when the recorded PID is still alive on this host (never requeue
beside a live worker). Surfaced via DispatchResult.reconciled_orphans.
2. Per-server MCP identity header (mcp_servers.<name>.identity_header,
config.yaml): optional {name, value_from: static|profile, value}
mapping; the header is attached to that server's HTTP/SSE transport
requests. 'static' sends the config value; 'profile' resolves the
active Hermes profile name once at connect time (no per-call
mutation). Explicit per-server headers of the same name (any casing)
win. Invalid blocks warn-and-ignore; stdio servers warn-and-ignore.
Tests: tests/gateway/test_kanban_reconcile_orphans.py (9),
tests/tools/test_mcp_identity_header.py (13), all written first (RED)
then implemented (GREEN). No new HERMES_* env vars.
Inspired by: openai/symphony tracker reconciliation (Apache-2.0) +
Poke per-user MCP identity (idea-level).
A task could already pin a model and provider, but not how hard the worker
thinks: reasoning effort came from the assigned profile's config and nothing
per-task could reach it. Pairing a small model with high effort, or a big one
with thinking off, meant editing the worker profile itself.
Adds a tasks.reasoning_effort column (migrated, NULL = inherit the profile)
with set_reasoning_effort(), a create_task kwarg, and a --reasoning spawn flag.
Kept deliberately independent of model_override: a task may run the profile's
own model at a different depth, and clearing a model override no longer resets
the depth the operator chose. "none" is a value (thinking off), not a clear.
--reasoning is new on the CLI too — the level was only reachable through the
/reasoning slash command, so the dispatcher had no flag to pass. It overrides
agent.reasoning_effort for one run and is never persisted.
The retag gate was global, so once one board reclaimed its legacy rows a
second board on the same state.db never got swept. Key the state_meta gate
on the workspaces root and skip reopening state.db on every spawn via an
in-process set. Align the dispatcher-spawn test with the worker's own
`kanban` source tag and cover the per-board gate.
Workers spawn as `hermes chat -q "work kanban task <id>"` without
HERMES_SESSION_SOURCE, so every attempt persisted as an untitled `cli`
row. Tag them `kanban` and register it as a local, non-messaging
surface.
A running worker now polls its comment thread and folds new operator notes
into the live turn via the OUT-OF-BAND steer channel (list_comments_after +
a heartbeat-driven bridge, watermarked so history isn't re-injected and the
worker's own notes are skipped). No block→comment→unblock dance. Desktop's
composer sends notes live ("delivered within a few seconds") with "Requeue
with note" as the restart option and a help tooltip.
Boards gain an optional project_id. When set, the board's default_workdir
mirrors the project's primary repo and every new task inherits the project —
a deterministic worktree + branch per task — unless it names its own. New
GET /projects; board create/patch/list carry project_id + resolved name; the
create dialog defaults its workspace to the board's and allows a per-task
path override. Desktop: "Board settings…" gains a project picker.
The WAL→DELETE fallback on WAL-incompatible filesystems (NFS / SMB / FUSE /
the AgentFS NFS overlay) was logged at WARNING, treating a real loss of
concurrency — under the kanban dispatcher + workers a write blocks readers,
surfacing as SQLITE_BUSY — as if it were cosmetic. Escalate the deduplicated
fallback log to ERROR so the degradation is observable, not silent.
Add an opt-in require_wal=True to apply_wal_with_fallback that raises a typed
WalUnsupportedError (subclass of sqlite3.OperationalError, so existing DB-init
handlers still catch it) instead of degrading to DELETE, for callers that
mandate WAL concurrency. All four current callers keep the default
require_wal=False so NFS-homed installs keep working unchanged.
Tests: 4 new require_wal cases; WARNING→ERROR assertion updates in both
test_hermes_state_wal_fallback.py and test_kanban_db.py.
Teknium