victor-kyriazakos
3683e70043
* feat(relay): live-card ops — native draft streaming + task cards over the relay (gateway half)
NS-658. Three additive ops within contract v1, emitted only when the
connector's negotiated descriptor advertises them:
{op: draft, chat_id, draft_id, content, final, metadata}
{op: task_card, chat_id, card_id, chunks, metadata}
{op: task_card_stop, chat_id, card_id, metadata}
The gateway side is deliberately dumb: no platform API knowledge, no new
config keys. Slack mechanics (chat.startStream/appendStream/stopStream,
per-workspace feature-gate cache, send+edit fallback) live connector-side
where the platform adapter lives in the relay model.
Semantic bridge: base send_draft is Telegram-shaped (draft clears; final
is a separate send). Slack native streaming makes the stream THE message.
The adapter tracks the open draft per chat and converts the turn-final
send() into draft(final=true) so the connector seals the stream instead
of posting a duplicate; the stream ts returns as the message identity.
A failed frame disarms interception so the edit-based fallback's real
send goes through untouched.
BEHAVIOR CHANGE (deliberate): relay supports_draft_streaming() now
requires the descriptor flag AND the draft op. Flag-only was a latent
lie — send_draft inherited NotImplementedError, so a connector setting
the flag without the op would have crashed the stream consumer's draft
path. supported_ops stays fail-open for legacy (pre-contract) ops;
draft/task_card did not exist pre-contract and must not fail open.
Task cards ride #85476's adapter-agnostic TurnRunner seam (hasattr on
send_native_task_card_progress); supports_native_task_cards() is the
descriptor probe. Connector half + E2E harness pair follow in the gg
repo.
* fix(relay): expose native_task_cards_enabled() on the relay adapter
Live-canary finding (Alice, staging): the TurnRunner's task-card lane
probes adapter.native_task_cards_enabled() (the native Slack adapter's
opt-in contract). The relay adapter only offered
supports_native_task_cards(), so the hasattr gate failed silently and
tool progress stayed on the text path — draft streaming worked, cards
never rendered. Alias it to the descriptor probe.
* fix(relay): match task-card methods to the TurnRunner's native keyword contract
Live-canary finding #2 (Alice, staging): gateway/run.py's card lane calls
send/stop_native_task_card_progress with the NATIVE Slack adapter's
signature (tasks/title/reply_to/metadata/fallback_text, keyword-only) —
PR 85796's relay methods took a positional card_id, so every call raised
TypeError('unexpected keyword argument reply_to') in the progress task,
repeatedly killing the card publisher (and the retry loop resent the
final delivery 4-5x). Card id now derives per turn thread
(turn:<reply_to>), thread_ts anchored like draft; title/fallback_text
accepted for parity, not forwarded (plan-mode stream renders chunks).
* fix(relay): one draft stream per turn for stream-is-the-message adapters
Live-canary finding #4 (Alice, staging): the stream consumer bumps
draft_id at every tool boundary so Telegram-shaped drafts animate each
text segment as a fresh preview. On relay Slack NATIVE streaming a new
draft_id opens a brand-new chat.startStream — the user saw one frozen
message per segment (stuck streaming cursor ▉, never sealed: only the
LAST stream gets the final=true seal) plus the real final; 5-6 cumulative
snapshots per turn. Adapters that mark draft_stream_is_message keep ONE
stream per turn: tool progress lives in the native task card, and the
connector's suffix-delta falls back to whole-text append on prefix
mismatch, so segments append cleanly. Telegram-shaped drafts keep the
per-segment bump.
* fix(relay): don't seal the native stream at tool boundaries — only the turn-final does
Live-canary finding #5 (Alice; supersedes the incomplete #4 which was
necessary but not sufficient). Root cause CONFIRMED by integration trace
(test_live_cards_flow_trace.py, real consumer semantics + real adapter +
stub transport): at every tool boundary the consumer calls
_send_or_edit(finalize=True), which skips the draft path and issues a
real send(); the relay adapter's seal-interception converts THAT into
draft(final=true) — sealing the stream once per segment. Timeline showed
3 seals for a 3-segment turn: exactly the frozen cumulative ▉ snapshots
seen live (the replaced stream never gets stopStream, keeping its cursor).
Fix: for draft_stream_is_message adapters, a segment-break finalize
(finalize=True, is_turn_final=False) stays ON the draft path as another
cumulative frame; only got_done (is_turn_final=True) falls through to
send() and seals. Telegram-shaped platforms unchanged. Trace test now
pins the invariant: ONE user-visible message per turn.
* fix(relay): strip the text cursor from native draft frames
Live-canary finding #6 (Alice) — the ACTUAL duplicate-content mechanism,
confirmed by full-flow scan of both sides' code + logs. The consumer
appends its text cursor (▉) to every non-final display_text tick. The
connector's stream sender diffs CUMULATIVE frames via prefix check:
'abc▉'.startsWith → 'abc def▉' is NEVER a prefix match (the cursor sits
mid-string), so deltaFor falls back to whole-text append on EVERY tick —
chat.appendStream stacks each full cumulative snapshot (cursor included)
into the ONE stream message. Exactly the observed thread: repeated
blocks, each ending in a frozen ▉, growing per tick.
Fixes #4/#5 were real (one stream per turn now) but this was the last
mechanism standing. Native streams render their own typing indicator, so
the text cursor is pure noise on this path: strip it from draft frames.
Prefix check now holds; every tick appends only its true suffix delta.
* fix(relay): seal-interception covers EVERY egress door, not just send()
Live-canary finding #7 (Alice): one duplication remained after #6 — the
stream froze mid-word with the live indicator (never sealed) and the
final posted as a separate message. Log receipt: 'Queued follow-up:
final text delivery confirmed; delivering explicit media before
continuing' — the turn's final went out via the DELIVERY RESOLVER lane
(gateway/delivery.py), which calls send_for_platform() DIRECTLY,
bypassing send() and its seal-interception. The open stream never
absorbed the final; it arrived as a plain 'send' op → chat.postMessage.
Fix: hoist the open-draft check to the top of send() (ahead of the
explicit-platform branch) AND add it to send_for_platform() — an open
native stream absorbs the turn-final regardless of which egress door it
arrives through. The stream IS the message.
* fix(relay): failed seal falls back to plain send (PR 85796 AI-review point 1)
A turn-final seal that fails at the transport must never swallow the
final answer: the stream consumer has already disabled the draft
transport for the run, so a failed _seal_open_draft returning
success=False meant the user got NOTHING. Both seal-interception sites
(send + send_for_platform) now fall through to the regular plain-send
path on seal failure, with a warning receipt. Also mitigates AI-review
point 2 (sticky _open_draft_by_chat after an abandoned turn): a stale
entry's failed seal no longer blocks the next turn's delivery.
* fix(relay): arm seal-interception optimistically; never disarm on ambiguous failure (audit G-D1)
Deep-audit defect G-D1 (HIGH): the outbound leg is at-most-once on the
wire but its ack channel is lossy — send_outbound timeout (30s) and
WS-drop 'failures' frequently mean the frame WAS delivered and the
connector stream is open. send_draft popped _open_draft_by_chat on any
failure, disarming seal-interception while the connector stream lived:
the turn-final went out as a plain send → orphaned mid-word stream +
complete duplicate final (intermittent; needs a drop/timeout inside the
draft window).
Fix: arm the entry BEFORE the transport call and keep it armed on
failure/exception. Safe in every case: sealing a non-existent stream
opens+seals a single complete message connector-side, and a truly failed
seal already falls back to plain send at both interception sites.
Stale-entry damage is self-healing (one warning + plain send).
* fix(relay): gateway-side sealed-draft tombstone — G-D1 arming must not resurrect sealed streams
Regression fix on G-D1 (live: 'worse than before' — escalating frozen
prefixes). Optimistic arming had no seal-awareness: a straggler frame
arriving AFTER the seal re-armed _open_draft_by_chat for the already-
sealed draft_id; the next send was converted to draft(final=true) on the
tombstoned connector key, which CLEARED the connector tombstone (final
frame = new-turn signal), re-opened a stream with cumulative content,
and left it frozen — repeating per straggler: 4-5 escalating frozen
snapshots. Mirror the connector: _sealed_draft_by_chat records the
sealed draft_id per chat (tombstoned BEFORE the seal's transport call);
send_draft for a sealed draft_id is a success no-op (content already in
the sealed message) and never arms. A new turn's fresh draft_id arms
normally.
* fix(relay): key stream/card state per (chat, turn anchor) — parallel turns must not collide (finding #10)
Live finding #10 (Alice; three concurrent turns in one flat DM): all
coordination state was keyed per CHAT on a one-active-turn assumption.
Three parallel turns produced: turn B's task card merged into turn A's
(both were card 'turn:root' — reply_to is None in flat DMs), B left
cardless, and _open/_sealed_draft_by_chat clobbered across writers (3x
duplicate finals on the last turn). Per-turn machinery was correct;
the keys were not.
Fix: _draft_key(chat, metadata) = chat + the turn's thread anchor
(inbound stamps thread_ts = event.thread_ts or ts on every top-level
message, so each turn has one even in flat DMs). draft arming, seal
tombstones, both interception sites, and the task-card id all derive
from the same anchor. New trace test pins two interleaved turns:
distinct cards, own-stream seals, no leaked plain send, no cross-turn
tombstone drops (289 tests green).
* fix(gateway): preserve cumulative native stream across tools
* fix(gateway): consumer-declared final — the seal carries the true final
Three composed fixes for the Slack live-cards duplicate-final class:
1. finish(final_text): TurnRunner passes the completed final_response
(verifier footer, completion explainer included) as the authoritative
finalize payload. The native-stream seal delivers the TRUE final, so
post-stream mutation no longer forks a corrective plain send (#11).
2. Interim-send contract: commentary and segment-tail sends carry a
gateway-internal _interim_send marker; relay seal-interception skips
them at both egress doors. A mid-turn interim send can no longer seal
the live stream and orphan the real final into a duplicate.
3. Queued-follow-up lane reconciles an unconfirmed final by EDITING the
consumer's delivered message in place (sealed stream = regular
message, chat.update live-verified); plain send only as fallback.
This was the actual duplicate lane in the parallel canaries — every
duplicated turn logged 'final stream delivery not confirmed; sending
first response' (subagent-completion queued inbound), not parallelism.
Also: draft frames stay prefix-stable gateway-side (no fence-closing, no
segment state reset, no commentary reset for stream-is-the-message
adapters; MagicMock-safe 'is True' guards).
* test+docs: streaming-contract coverage completeness + maintenance guidelines
Coverage: two gaps closed on the consumer-declared-final contract —
(1) send_for_platform (the delivery-resolver egress door) honors the
_interim_send contract: no seal, marker stripped before the wire;
(2) finish(final_text) on a turn that never streamed does not adopt the
final (delivery ownership stays with the gateway's normal send path for
non-streaming models / tool-only turns).
Docs: AGENTS.md 'Known Pitfalls' gains the streaming delivery contract —
the four invariants of stream-is-the-message adapters (prefix-stable
frames, consumer-declared final, interim-send marker, reconcile-by-edit),
each traced to its live incident, plus the live-probed Slack streaming
API ground truth and the MagicMock 'is True' guard-style note.
* fix(relay): seal transport failure must never silently lose the final (review B1)
Two halves of one silent-loss path, live-probed on the review branch:
1. adapter: _seal_open_draft did not catch transport exceptions. A socket
drop at seal time raised out of send(), skipping the fail-open plain
send entirely. Now: retry the SAME idempotent final frame once (the
connector's sealed-key tombstone returns the original stream ts for a
repeated final — a retry can never open a second stream or duplicate),
then report failure so the caller's fail-open path runs.
2. consumer: the turn-final retry (elif not _already_sent) called
_send_or_edit with finalize=False, which re-entered the DRAFT-FRAME
branch. Its no-op dedupe compared the adopted final against the last
unsealed frame, matched, and returned True with ZERO transport calls —
final_response_sent went green, delivered_final_matches reconciled,
the gateway suppressed its fallback, and the user never received the
answer. finalize=True keeps this retry out of the draft branch.
Regression suite: tests/gateway/test_relay_seal_failure.py (3 tests).
Mutation evidence in follow-up verification: reverting either half sends
the suite red.
* fix(relay): draft ids unique across gateway incarnations (review B3)
The relay connector tombstones sealed streams by (channel, draft_id) and
keeps up to 512 of them; they outlive the gateway process. Relay gateways
are disposable BY DESIGN (scale-to-zero), and _draft_id_counter restarted
at zero every incarnation — so the first turns after every scale-from-zero
in a recently-active channel replayed already-sealed wire identities. The
connector answered those frames straight out of the old tombstone: zero
Slack API calls, the OLD message ts returned as the new turn's identity,
the new answer silently dropped while gateway-side flags recorded success.
Seed the counter from wall-clock milliseconds at process start. Ids stay
plain ints within the existing contract op; incarnations cannot overlap
for realistic turn counts and restart gaps.
Regression: tests/gateway/test_draft_id_restart_uniqueness.py — the seed
test fails on the old code (seed 0 is not epoch-scale).
* fix(relay): stream/card state keyed per TURN, not per thread anchor (review B2)
The thread anchor is the wrong coordination identity — simultaneously:
- too coarse: two parallel turns replying INSIDE ONE Slack thread share
thread_ts. Live-probed on the review branch: turn A's final sealed turn
B's stream with A's content while A's own stream stayed open, and B's
final degraded to a plain send.
- too fragile: a flat DM with no thread metadata degraded to the bare
chat id, re-creating the original finding-#10 collision the anchor was
meant to fix.
_draft_key now prefers the triggering inbound message id (message_id /
reply_to_message_id — per-turn by construction; the gateway's Slack
thread metadata and the consumer's send path both stamp it), falling back
to the thread anchor, then the bare chat. The consumer stamps the same
reply_to_message_id on draft frames so frames and the turn-final resolve
to one key. Task-card ids share the derivation via _card_key (one helper
for send AND stop, so the stop always hits the stream the send opened).
Legacy resolver-lane callers with placement-only metadata still seal via
_match_open_draft's fallback — but ONLY when exactly one stream is open.
With several open, an identity-less send stays a plain send: a duplicate
message is recoverable, sealing someone else's stream is not.
Regression: tests/gateway/relay/test_relay_turn_keying.py (7 tests).
* fix(relay): stream-is-the-message is a Slack semantic, gate it on the descriptor (review B4)
draft_stream_is_message was hardcoded True on the relay adapter class,
i.e. for EVERY relay platform. The base send_draft contract is
Telegram-shaped — the draft clears client-side and the final arrives as
a separate real send that becomes the history message. With the flag
forced on, any non-Slack connector advertising the draft op had its
turn-final intercepted into draft(final=true): probed on the review
branch with a telegram descriptor, the op stream was
[draft(final=false), draft(final=true)] and NO send — no history message
would ever be posted.
Gate the flag on the negotiated descriptor platform (slack), and skip
arming seal-interception entirely when it is off. A future platform with
genuine stream-is-the-message native streaming should advertise it via
the descriptor rather than widening the platform check by guesswork.
Regression: tests/gateway/relay/test_relay_stream_semantics_gating.py
(4 tests: gating both ways, telegram final is a real send, slack final
still seals).
* fix(gateway): mark every mid-turn status lane interim — heartbeats must not seal the stream (review B5)
Seal-interception treats the first unmarked send to an armed (chat, turn)
key as the turn-final. The consumer's own interim lanes (commentary, tail
flush) carry _interim_send, but four gateway-side lanes that fire DURING
a streaming turn did not:
- long-running heartbeat (default every 180s — probed live: at 3 minutes
it sealed the live stream with '⏳ Working — 3 min', the real final
posted as a duplicate, and later frames were silently swallowed by the
seal tombstone)
- inactivity warning
- plain-text approval fallback (button lane failed)
- background-review notice
Add _interim_metadata() beside _non_conversational_metadata and wrap all
four call sites. The marker is gateway-internal; the relay adapter strips
it before the wire (existing behavior, pinned by test).
Note for follow-up: the opt-out shape remains fragile — any FUTURE
unmarked mid-turn send lane re-creates this bug. Inverting the contract
(explicitly mark the one turn-final send) is the durable fix but touches
every adapter's final-delivery path; deliberately kept out of this
review-fix series.
Regression: tests/gateway/test_interim_send_lanes.py (4 tests).
* fix(gateway): interrupted/incomplete turns must not adopt the diagnostic as the stream final (review B6)
The finish(final_text) adoption gate checked only 'not failed', but the
interrupt/abort returns in agent/conversation_loop.py are
{completed: False, interrupted: True, final_response: 'Operation
interrupted during …'} with NO failed key. Adopting that diagnostic:
1. sealed the user's streamed partial answer over with the interrupt
text (stream-is-the-message: the seal rewrites the whole message), and
2. recorded the diagnostic as the turn-final payload, so
delivered_final_matches reconciled and the gateway suppressed its own
error-delivery path — the diagnostic became the ONLY thing delivered.
Enumerated all 27 final_response-bearing return shapes in
conversation_loop.py: every non-happy-path shape carries completed:
False (several with a diagnostic final_response and neither failed nor
interrupted — retry exhaustion, truncation, codex-incomplete); the happy
path routes through turn_finalizer.finalize_turn (completed=True). Gate
is therefore: not failed AND not interrupted AND completed is not False.
Results lacking the completed key entirely (older callers/test doubles)
keep the previous behavior.
Regression: tests/gateway/test_stream_final_adoption_gate.py (6 tests,
incl. a source-level pin on the run.py call site).
* fix(relay): task-card transport failures degrade to failed SendResults (review B7)
send_native_task_card_progress and stop_native_task_card_progress let
transport exceptions escape. The stop runs inside the progress loop's
finally block on the turn-cleanup path, and the post-cancel awaits in
gateway/run.py caught only CancelledError — a socket drop during a card
publish/stop therefore aborted cleanup BEFORE the final-delivery
bookkeeping ran.
Three layers, outermost defends any adapter:
- both adapter methods catch transport exceptions and return failed
SendResults (progress is advisory; the TurnRunner's text fallback
already handles failure results)
- the progress loop's finally wraps the stop (best-effort; the connector
seals orphaned card streams on its own via recycling/eviction)
- the cleanup awaits log-and-continue on non-cancellation errors so
final-delivery bookkeeping always runs
Regression: tests/gateway/relay/test_relay_task_card_failures.py.
* fix(relay): a dying turn seals its native stream instead of orphaning it (review B8)
Stale-generation exits (/new, /stop mid-stream) and cancellations
returned from the consumer's run() with the native stream still open:
- the Slack message kept its live streaming indicator forever (the
cancellation best-effort edit only runs when _message_id exists, and
the native draft path deliberately keeps it None);
- the adapter's armed interception state survived the turn, so the next
turn on the same key could inherit it and seal a dead draft_id.
New adapter op abandon_open_draft(chat, content): seals in place with
the text already on screen (the consumer passes its last delivered
frame) — the seal adds nothing and claims nothing; delivery flags are
never set, so the gateway's normal paths still own whatever happens
next. Best-effort by contract (failure reported, never raised); the
connector reaps truly orphaned streams via recycling/eviction.
The consumer calls it from both death paths: the stale-generation early
return and the CancelledError handler.
Regression: tests/gateway/test_stream_abandon_on_turn_death.py (4 tests,
incl. the next-turn-inheritance hazard).
* fix(relay): bound the draft/seal coordination dicts (review M1)
_sealed_draft_by_chat's key embeds a per-turn identity, so every
completed turn wrote a permanent entry — unbounded growth for the life
of a long-running gateway process (the docstring said 'one entry per
chat', which stopped being true when the key gained the turn anchor).
_open_draft_by_chat could grow the same way via abandoned entries.
FIFO-evict both at 512 entries — the same idiom as the sibling bounded
cache (_auto_thread_by_chat, capped at 256) and the same size as the
connector's own tombstone store. The straggler window the tombstone
exists for is seconds long; FIFO is more than enough.
Regression: tests/gateway/relay/test_relay_state_bounds.py.
* fix(relay): explicit connector rejection disarms interception; exceptions stay armed (review P3)
The G-D1 optimistic-arming change silently dropped disarm-on-failure
entirely: after an EXPLICIT connector rejection (success=False result —
not a transport ambiguity), interception stayed armed even though the
stream consumer disables the draft transport on that failure and falls
back to edit-based streaming. Its turn-final would then be converted
into a seal on a stream the connector just told us is unusable.
test_draft_failure_result_propagates claimed to cover this ('must NOT
leave seal-interception armed') but passed for an unrelated reason: the
stub's canned failure also failed the SEAL, whose fail-open path did the
plain send.
Split the two semantics and pin each honestly:
- explicit rejection (result success=False): disarm — turn-final is a
real send (test_draft_failure_result_propagates, now testing what its
comment says)
- transport exception: ambiguous, stay armed — turn-final still seals
(test_draft_transport_exception_keeps_interception_armed, the G-D1
contract)
Also corrects commit ba3a24a's claim ('a failed frame disarms
interception so the edit-based fallback's real send goes through
untouched') to hold again for the rejection case it described.
* fix(relay): lost acks are ambiguous, not rejections — on the RESULT channel too (review r2, finding 1)
The production ws transport does not raise on ack timeout — it returns
{"success": False, "error": "relay outbound timed out"}. The round-1
ambiguity handling keyed entirely on the exception channel, so the shape
production actually produces was misclassified as a definite connector
rejection. Probed on the head:
- lost SEAL ack: skipped the idempotent retry, fell straight to a plain
send — duplicate final whenever the seal had actually applied;
- lost FRAME ack: the round-1 disarm-on-rejection fired — interception
disarmed, frozen native stream beside a plain final. This re-created
the original G-D1 ambiguous-ack defect on the result channel.
Contract now spans both channels:
- transport: the ack-timeout branch tags ambiguous=True. The fail-fast
branches (closing / not connected) never sent anything and stay
unmarked — they are definite non-delivery.
- adapter frame path: ambiguous results keep interception armed (same
as exceptions); only definite rejections disarm.
- adapter seal path: one shared _attempt() classifier — exception and
ambiguous result both mean "unknown"; the SAME idempotent frame is
retried once (connector tombstone returns the original stream ts for
a repeated final). Only after both attempts stay ambiguous does the
caller's fail-open plain send run: a possible duplicate after double
ack loss beats a silent loss, and double ack loss on one socket
almost always means the transport is down for the plain send too.
Regression: tests/gateway/relay/test_relay_ack_ambiguity.py (6 tests,
incl. a source-of-truth check that the transport tags the timeout branch
and leaves fail-fast branches unmarked).
* fix(relay): stream semantics + draft capability resolve per CHAT, not per primary (review r2, finding 2)
One RelayAdapter fronts N platforms (Phase 1.5): descriptors accumulate
per platform on the transport and egress is tagged per chat — but the
round-1 gate keyed draft_stream_is_message and supports_draft_streaming()
off the PRIMARY scalar descriptor. Probed on the head:
- Slack primary + Telegram chat: the Telegram chat's turn-final was
intercepted into draft(final=true) — no real Telegram history message;
- Telegram primary + Slack chat: the Slack chat was denied native
streaming entirely.
Resolve both through _descriptor_for_chat — the same per-chat machinery
max_message_length already uses (added for the identical class of bug:
the primary's 39000-char cap over-sending into Discord 400s):
- new stream_is_message_for_chat(chat_id) on the adapter; arming and
NotImplementedError gating use it. The class attribute remains as the
single-platform value and legacy-probe fallback.
- supports_draft_streaming() gains an optional chat_id kwarg (base
signature updated; single-platform adapters ignore it). The consumer
passes chat_id with a TypeError fallback for out-of-tree adapters.
- the consumer's four draft_stream_is_message reads collapse into one
_stream_is_message() helper that prefers the per-chat probe
(class-resolved, MagicMock-safe) over the attribute.
Platform-name inference ("slack") stays deliberate: a descriptor-level
semantic field is the right eventual contract but is a cross-repo wire
change — noted for the gg follow-up so future platforms advertise the
semantic explicitly.
Regression: tests/gateway/relay/test_relay_multiplatform_semantics.py
(5 tests: both starvation directions, scalar fallback, per-chat
capability gate).
* fix(gateway): split delivery + authoritative footer reconciles by suffix, not full resend (review r2, finding 3)
The _FINAL_TEXT adoption guard refuses wholesale adoption on split turns
— correct (#78541: sealed heads would repeat inside the tail) but it was
absolute: a post-split verifier footer never entered the ledger,
delivered_final_matches() reported a mismatch, and the gateway resent
the ENTIRE body+footer after the split chunks (the #11 duplicate class,
one level up).
When the authoritative final strictly prefix-extends the split ledger,
the missing suffix is the only undelivered content: append it to the
live tail and the ledger, so the finalize carries it and the recorded
payload reconciles. Non-prefix rewrites keep the full-resend fallback —
a rewrite cannot be patched onto sealed heads.
Regression: tests/gateway/test_split_final_suffix_reconcile.py (3 tests:
suffix rides the tail + reconciles, rewrite still mismatches, unsplit
adoption unchanged).
* fix(relay): cancellation mid-seal restores open state so abandon can close the stream (review r2, finding 4)
_seal_open_draft pops the open entry and writes the local tombstone
BEFORE awaiting transport I/O — correct ordering for the straggler race,
but CancelledError is not an Exception: a cancel during the await
bypassed all failure handling, leaving the remote stream live (visible
streaming indicator until connector eviction) while the local state said
'nothing open'. The consumer's abandon pass — added for exactly this
turn-death case — found nothing to close and no-oped.
On CancelledError: restore the open entry, drop the premature tombstone
(only if it is still ours), re-raise. The abandon path then seals the
stream in place with the on-screen text.
Regression: tests/gateway/relay/test_relay_seal_cancellation.py (2
tests: state restoration, and end-to-end cancel→abandon→remote seal).
* fix(relay): thread anchors are placement, not turn identity — revive the placement-only fallback (review r2, finding 5)
_match_open_draft's single-open-stream fallback was dead for its primary
intended callers: metadata carrying thread_ts/thread_id (placement-only
resolver lanes) was classified as having 'turn identity', so those sends
never reached the fallback — probed: a plain final posted beside the
still-open turn-keyed stream.
Only per-turn MESSAGE ids are identity now. Thread-anchored and bare
callers share the fallback: absorb into the chat's open stream when
EXACTLY one is open; stay a plain send when several are (duplicate is
recoverable, wrong-stream seal is not). Callers WITH a message id whose
key misses never fall back — their identity is authoritative and a miss
means the stream belongs to a different turn.
Regression: 4 new tests in test_relay_turn_keying.py (thread-anchored
seal, both ambiguous-stay-plain shapes, id-mismatch never steals).
* fix(relay): random process nonce for draft-id seeding (review r2, follow-up 6)
The epoch-millisecond seed (round-1 B3 fix) mitigates the restart-replay
class but is not a uniqueness guarantee: two gateways starting in the
same millisecond, a forked process inheriting the class state, or a
clock step backwards can all mint colliding wire identities against the
connector's per-(channel, draft_id) tombstone store.
Seed from secrets.randbits(49) instead: collision probability negligible,
no clock dependence, and ids + realistic per-process turn counts stay
comfortably inside the connector's JS number range (draft_id?: number,
2^53). Regression test now spawns two real interpreters and asserts
their seeds differ — the exact scale-to-zero restart shape, and both
start within the same second so a clock-locked seed would fail it.
* fix(relay): stamp per-turn Slack egress identity — cache is fallback only (R3-5)
The connector (gateway-gateway#210) fills chat.startStream's
recipient_user_id / recipient_team_id — required by Slack when
streaming to a channel — from metadata.user_id / metadata.scope_id.
The gateway stamped only slack_team_id per-turn and left user_id (and
scope_id) to RelayAdapter._with_scope, whose per-chat caches are keyed
on chat_id alone and overwritten by every inbound message: with users
U1 and U2 running overlapping turns in one channel, U2's arrival
overwrote the cache before U1's stream opened, and U1's stream carried
U2 as recipient_user_id.
_thread_metadata_for_source now stamps scope_id and user_id from the
turn's OWN source (setdefault — explicit values win), so identity is
turn-scoped data on the wire. _with_scope is unchanged and fill-only:
the caches keep serving restart/synthetic sends that carry no per-turn
identity, which is all they were ever safe for.
Mutation evidence: reverting the run.py hunk sends
test_thread_metadata_stamps_per_turn_user_and_scope and
test_concurrent_turns_carry_their_own_identity red; restore returns
green. The _with_scope fill-only tests pass on both trees (existing
correct behavior, now pinned against regression).
---------
Co-authored-by: Ben Barclay <ben@nousresearch.com>
1609 lines
84 KiB
Markdown
1609 lines
84 KiB
Markdown
# Hermes Agent - Development Guide
|
||
|
||
Instructions for AI coding assistants and developers working on the hermes-agent codebase.
|
||
|
||
**Never give up on the right solution.**
|
||
|
||
## What Hermes Is
|
||
|
||
Hermes is a personal AI agent that runs the same agent core across a CLI, a
|
||
messaging gateway (Telegram, Discord, Slack, and ~20 other platforms), a TUI,
|
||
and an Electron desktop app. It learns across sessions (memory + skills),
|
||
delegates to subagents, runs scheduled jobs, and drives a real terminal and
|
||
browser. It is extended primarily through **plugins and skills**, not by
|
||
growing the core.
|
||
|
||
Two properties shape almost every design decision and are the lens for
|
||
reviewing any change:
|
||
|
||
- **Per-conversation prompt caching is sacred.** A long-lived conversation
|
||
reuses a cached prefix every turn. Anything that mutates past context,
|
||
swaps toolsets, or rebuilds the system prompt mid-conversation invalidates
|
||
that cache and multiplies the user's cost. We do not do it (the one
|
||
exception is context compression).
|
||
- **The core is a narrow waist; capability lives at the edges.** Every model
|
||
tool we add is sent on every API call, so the bar for a new *core* tool is
|
||
high. Most new capability should arrive as a CLI command + skill, a
|
||
service-gated tool, or a plugin — not as core surface.
|
||
|
||
## Contribution Rubric — What We Want / What We Don't
|
||
|
||
This is the project's intent layer. Use it two ways:
|
||
|
||
1. **For humans and for your own work** — what gets merged and what gets
|
||
rejected, so a contribution aims at the target.
|
||
2. **For automated review (the triage sweeper)** — guidance on when a PR is
|
||
safe to close on the three allowed reasons (`implemented_on_main`,
|
||
`cannot_reproduce`, `incoherent`) and, just as important, **when NOT to
|
||
close** one. Taste-based "we don't want this / out of scope" closes are NOT
|
||
an automated decision — those stay with a human maintainer. The sweeper's
|
||
job here is to recognize design intent and *avoid wrongly closing a
|
||
legitimate contribution*, not to make the won't-implement call itself.
|
||
|
||
Read the balance right: Hermes ships a **lot** — most merges are bug fixes to
|
||
real reported behavior, and the product surface (platforms, channels,
|
||
providers, models, desktop/TUI features) expands aggressively and on purpose.
|
||
The restraint below is aimed squarely at the **core agent + the model tool
|
||
schema**, the one place where every addition is paid for on every API call.
|
||
"Smallest footprint" governs *how a capability is wired into the core*, NOT
|
||
whether the product is allowed to grow. We are expansive at the edges and
|
||
conservative at the waist.
|
||
|
||
### What we want
|
||
|
||
- **Fix real bugs, well.** The bulk of what lands is `fix(...)` against an
|
||
actual reported symptom. A good fix reproduces the symptom on current
|
||
`main`, points to the exact line where it manifests, and fixes the whole bug
|
||
class — sibling call paths included — not just the one site the reporter hit.
|
||
- **Expand reach at the edges.** New platform adapters, channels, providers,
|
||
models, and desktop/TUI/dashboard features are welcome and land routinely,
|
||
including large ones (a new messaging channel, a session-cap feature, a
|
||
Windows PTY bridge). Breadth in the product is a goal, not a footprint
|
||
concern — as long as it integrates with the existing setup/config UX
|
||
(`hermes tools`, `hermes setup`, auto-install) rather than bolting on a raw
|
||
env var.
|
||
- **Refactor god-files into clean modules.** Extracting a multi-thousand-line
|
||
cluster out of `cli.py` / `run_agent.py` / `gateway/run.py` into a focused
|
||
mixin or module is wanted work, even when the diff is huge and mechanical
|
||
(large `+N/-N` refactors merge regularly). The "every line traces to the
|
||
request" test applies to *feature* PRs; a declared refactor's request IS the
|
||
extraction.
|
||
- **Keep the core narrow.** New *model tools* are the expensive exception —
|
||
every tool ships on every API call. Prefer, in order: extend existing code →
|
||
CLI command + skill → service-gated tool (`check_fn`) → plugin → MCP server
|
||
in the catalog → new core tool (last resort). See "The Footprint Ladder."
|
||
- **Extend, don't duplicate.** Before adding a module/manager/hook, check
|
||
whether existing infrastructure already covers the use case. When several PRs
|
||
integrate the same *category*, design one shared interface instead of merging
|
||
them one at a time (see the ABC + orchestrator note under the Footprint
|
||
Ladder).
|
||
- **Behavior contracts over snapshots.** Tests should assert how two pieces of
|
||
data must relate (invariants), not freeze a current value (model lists,
|
||
config version literals, enumeration counts). See "Don't write
|
||
change-detector tests."
|
||
- **E2E validation, not just green unit mocks.** For anything touching
|
||
resolution chains, config propagation, security boundaries, remote
|
||
backends, or file/network I/O, exercise the real path with real imports
|
||
against a temp `HERMES_HOME`. Mocks hide integration bugs.
|
||
- **Cache-, alternation-, and invariant-safe.** Preserve prompt caching, strict
|
||
message role alternation (never two same-role messages in a row; never a
|
||
synthetic user message injected mid-loop), and a system prompt that is
|
||
byte-stable for the life of a conversation.
|
||
- **Contributor credit preserved.** Salvage external work by cherry-picking
|
||
(rebase-merge) so authorship survives in git history; don't reimplement from
|
||
scratch when you can build on top.
|
||
|
||
### What we don't want (rejected even when well-built)
|
||
|
||
- **Speculative infrastructure.** Hooks, callbacks, or extension points with no
|
||
concrete consumer. Adding a hook is easy; removing one after plugins depend
|
||
on it is hard. A hook is NOT speculative if a contributor has a real, stated
|
||
use case — even if the consumer ships separately.
|
||
- **New `HERMES_*` env vars for non-secret config.** `.env` is for secrets
|
||
only (API keys, tokens, passwords). All behavioral settings — timeouts,
|
||
thresholds, feature flags, display prefs — go in `config.yaml`. Bridge to an
|
||
internal env var if the mechanism needs one, but user-facing docs point to
|
||
`config.yaml`. Reject PRs that tell users to "set X in your .env" unless X
|
||
is a credential.
|
||
- **A new core tool when terminal + file already do the job, or when a skill
|
||
would.** If the only barrier is file visibility on a remote backend, fix the
|
||
mount, not the toolset.
|
||
- **Lazy-reading escape hatches on instructional tools.** No `offset`/`limit`
|
||
pagination on tools that load content the agent must read fully (skills,
|
||
prompts, playbooks). Models will read page 1 and skip the rest.
|
||
- **"Fixes" that destroy the feature they secure.** A mitigation that kills the
|
||
feature's purpose is the wrong mitigation. Read the original commit's intent
|
||
(`git log -p -S`) before restricting behavior; find a fix that preserves the
|
||
feature.
|
||
- **Outbound telemetry / usage attribution without opt-in gating.** No new
|
||
analytics, third-party identifier tagging, or attribution tags until a
|
||
generic user-facing opt-in (config gate + setup prompt + `hermes tools`
|
||
toggle) exists. Park behind a label, do not merge.
|
||
- **Change-detector tests, cache-breaking mid-conversation, dead code wired in
|
||
without E2E proof, and plugins that touch core files.** Plugins live in their
|
||
own directory and work within the ABCs/hooks we provide; if a plugin needs
|
||
more, widen the generic plugin surface, don't special-case it in core.
|
||
- **Third-party products / other people's projects integrated into the core
|
||
tree.** Observability backends, vendor SaaS integrations, analytics dashboards,
|
||
and similar "someone else's product" plugins do NOT land under `plugins/` in
|
||
this repo. They place an ongoing maintenance burden on us to keep them working
|
||
against a fast-moving core, for a backend we don't own. Ship them as a
|
||
**standalone plugin repo** users install into `~/.hermes/plugins/` (or via a
|
||
pip entry point), and promote them in the Nous Research Discord
|
||
(`#plugins-skills-and-skins`). This is a coupling-and-maintenance decision, not
|
||
a quality bar — the plugin can be excellent and still be a close. PRs that add
|
||
such a directory to the tree are closed with a pointer to publish it as its own
|
||
repo.
|
||
|
||
### Before you call it a bug — verify the premise (and when NOT to close)
|
||
|
||
The most common reason a well-written PR gets closed is not code quality — it
|
||
is that the change is built on a **wrong premise**, or it treats an
|
||
**intentional design as a gap**. These patterns cut both ways: they tell a
|
||
human reviewer what to scrutinize, and they tell the automated sweeper when a
|
||
PR is NOT safe to close as `implemented_on_main` / `cannot_reproduce` (when in
|
||
doubt, leave it open for a human). They are distilled from real closes.
|
||
|
||
- **"Intentional design, not a gap."** A limitation that looks like an
|
||
oversight is often deliberate. Before "fixing" a missing link or a
|
||
restriction, ask whether the isolation IS the design. Example: profiles are
|
||
independent islands on purpose — a PR adding live config inheritance from the
|
||
default profile was closed because coupling profiles together is exactly what
|
||
the design prevents (the copy-at-creation `--clone` path already covers the
|
||
legitimate "start from my default" case). Read the original commit's intent
|
||
(`git log -p -S "<symbol>"`) before assuming something is unfinished.
|
||
- **"The premise doesn't hold against how X actually works."** A PR's
|
||
justification frequently rests on a wrong mental model of an existing
|
||
mechanism. Trace the real code/runtime before accepting the rationale. Two
|
||
real closes: a rate-limit "re-probe during cooldown" PR (the breaker only
|
||
trips on a *confirmed-empty* account bucket, so re-probing just hammers a
|
||
bucket we've already proven empty); a usage-accumulation fix whose new branch
|
||
**never executes at runtime** because an earlier guard already popped the
|
||
state it depended on. If you can't point to the exact line where the bug
|
||
manifests AND show the fix changes that line's behavior, you haven't verified
|
||
the premise.
|
||
- **"This fix was wrong — the absence/omission was deliberate."** Adding the
|
||
obvious-looking missing piece can break things the omission was protecting.
|
||
Example: restoring "missing" `__init__.py` files made a test tree importable
|
||
as a dotted package that shadowed the real plugin, deleting its `register()`
|
||
at import time. The absence was load-bearing.
|
||
- **"Overreached / resurrected an approach we'd moved past."** Scope creep that
|
||
supersedes an agreed-on base, or revives a direction the maintainers
|
||
deliberately closed, gets rejected even when the code works. Keep the change
|
||
to the narrow piece that was actually agreed; offer the rest as a focused
|
||
follow-up.
|
||
|
||
The throughline: **verify the claim AND the intent against the codebase before
|
||
writing or merging a fix.** A confirmed reproduction on current `main` plus a
|
||
line-level account of where the fix acts beats a plausible-sounding rationale
|
||
every time. When in doubt about intent, it is cheaper to ask than to ship a
|
||
fix that fights the design.
|
||
|
||
### The Footprint Ladder (new capability decision)
|
||
|
||
Each rung adds more permanent surface than the one above. Choose the highest
|
||
(least-footprint) rung that correctly solves the problem:
|
||
|
||
1. **Extend existing code** — the capability is a variation of something that
|
||
already exists. Zero new surface.
|
||
2. **CLI command + skill** — manages config/state/infra expressible as shell
|
||
commands. The agent runs `hermes <subcommand>` guided by a skill. Zero
|
||
model-tool footprint. Default choice for subscriptions, scheduled tasks,
|
||
service setup. Examples: `hermes webhook`, `hermes cron`, `hermes tools`.
|
||
3. **Service-gated tool (`check_fn`)** — needs structured params/returns AND
|
||
only appears when a prerequisite is configured. Zero footprint otherwise.
|
||
Examples: Home Assistant tools (gated on token), memory-provider tools.
|
||
4. **Plugin** — third-party/niche/user-specific capability that doesn't ship in
|
||
core. Lives in `~/.hermes/plugins/` or a pip package, discovered at runtime.
|
||
5. **MCP server (in the catalog)** — if the capability genuinely needs to be a
|
||
tool (structured I/O the agent invokes) but isn't core-fundamental, prefer
|
||
building it as an MCP server and adding it to the MCP catalog over growing
|
||
the core toolset. The agent connects to it through the built-in MCP client;
|
||
zero permanent core-schema footprint, and it's reusable by any MCP host.
|
||
6. **New core tool** — only when the capability is fundamental, broadly useful
|
||
to nearly every user, and unreachable via terminal + file (or an MCP server).
|
||
Examples of correct core tools: terminal, read_file, web_search,
|
||
browser_navigate.
|
||
|
||
When 3+ open PRs try to integrate the same *category* of thing (memory
|
||
backends, providers, notifiers), don't merge them one at a time — design an
|
||
ABC + orchestrator, wrap the existing built-in as the first provider, and turn
|
||
the competing PRs into plugins against that interface.
|
||
|
||
### Surface capability is a property of the SESSION, never of the process env
|
||
|
||
A tool that only works because of *who is on the other end of the connection* —
|
||
the desktop app's panes, the in-app browser, message reactions, Projects — must
|
||
resolve its availability from the **session's own source**, not from an env var
|
||
on the backend process.
|
||
|
||
The client and the backend are separate machines on separate clocks. The
|
||
desktop app can be driving a backend Electron spawned locally, one over SSH,
|
||
one behind a plain URL + token, or Hermes Cloud. Only the first two are spawned
|
||
by us and carry `HERMES_DESKTOP=1`. Every env-keyed GUI gate is therefore a
|
||
silent no-op on the other half of the topologies, and the failure is invisible:
|
||
the tool is stripped from the schema before the model ever sees it, on the same
|
||
backend whose platform hint is telling the model it's *"chatting inside the
|
||
Hermes desktop app."*
|
||
|
||
The pattern that works:
|
||
|
||
- **The toolset is the surface gate.** Keep the tools off `_HERMES_CORE_TOOLS`
|
||
(nobody else should pay their schema) and put them in a named toolset —
|
||
`desktop_ui`, `project`. The GUI gateway's `_load_enabled_toolsets(platform)`
|
||
folds that toolset in when the session's platform says GUI. One resolver,
|
||
every topology.
|
||
- **`check_fn` answers reachability or user opt-in, not surface.** "Is the
|
||
renderer bridge wired?", "did the user enable reactions?" — fine. "Was I
|
||
spawned by Electron?" — not fine. `check_fn` results are also TTL-cached
|
||
process-wide (`tools/registry.py`), so a per-session answer does not belong
|
||
there at all: one process serves many sessions.
|
||
- **Ask which identity you actually mean.** `HERMES_DESKTOP=1` legitimately
|
||
marks *"this backend process was spawned by the app"* — it gates the cron
|
||
ticker and web-dist handling correctly. It does NOT mean "a GUI is watching",
|
||
and the embedded terminal pane (`hermes --tui` against that same backend) is
|
||
the standing counterexample.
|
||
|
||
Same test both ways: if the capability would still make sense with the client
|
||
on another machine, it is session-scoped. Cover it with a test that asserts the
|
||
GUI session gets the tool **with the env var absent** — that's the assertion
|
||
the original gate could never have passed.
|
||
|
||
## Development Environment
|
||
|
||
```bash
|
||
# Prefer .venv; fall back to venv if that's what your checkout has.
|
||
source .venv/bin/activate # or: source venv/bin/activate
|
||
```
|
||
|
||
`scripts/run_tests.sh` probes `.venv` first, then `venv`, then
|
||
`$HOME/.hermes/hermes-agent/venv` (for worktrees that share a venv with the
|
||
main checkout).
|
||
|
||
## Project Structure
|
||
|
||
File counts shift constantly — don't treat the tree below as exhaustive.
|
||
The canonical source is the filesystem. The notes call out the load-bearing
|
||
entry points you'll actually edit.
|
||
|
||
```
|
||
hermes-agent/
|
||
├── run_agent.py # AIAgent class — core conversation loop (~12k LOC)
|
||
├── model_tools.py # Tool orchestration, discover_builtin_tools(), handle_function_call()
|
||
├── toolsets.py # Toolset definitions, _HERMES_CORE_TOOLS list
|
||
├── cli.py # HermesCLI class — interactive CLI orchestrator (~11k LOC)
|
||
├── hermes_state.py # SessionDB — SQLite session store (FTS5 search)
|
||
├── hermes_constants.py # get_hermes_home(), display_hermes_home() — profile-aware paths
|
||
├── hermes_logging.py # setup_logging() — agent.log / errors.log / gateway.log (profile-aware)
|
||
├── batch_runner.py # Parallel batch processing
|
||
├── agent/ # Agent internals (provider adapters, memory, caching, compression, etc.)
|
||
├── hermes_cli/ # CLI subcommands, setup wizard, plugins loader, skin engine
|
||
├── tools/ # Tool implementations — auto-discovered via tools/registry.py
|
||
│ └── environments/ # Terminal backends (local, docker, ssh, modal, daytona, singularity)
|
||
├── gateway/ # Messaging gateway — run.py + session.py + platforms/
|
||
│ ├── platforms/ # Adapter per platform (telegram, discord, slack, whatsapp,
|
||
│ │ # homeassistant, signal, matrix, mattermost, email, sms,
|
||
│ │ # dingtalk, wecom, weixin, feishu, qqbot, bluebubbles,
|
||
│ │ # yuanbao, webhook, api_server, ...). See ADDING_A_PLATFORM.md.
|
||
│ └── builtin_hooks/ # Extension point for always-registered gateway hooks (none shipped)
|
||
├── plugins/ # Plugin system (see "Plugins" section below)
|
||
│ ├── memory/ # Memory-provider plugins (honcho, mem0, supermemory, ...)
|
||
│ ├── context_engine/ # Context-engine plugins
|
||
│ ├── model-providers/ # Inference backend plugins (openrouter, anthropic, gmi, ...)
|
||
│ ├── kanban/ # Multi-agent board dispatcher + worker plugin
|
||
│ ├── hermes-achievements/ # Gamified achievement tracking
|
||
│ ├── observability/ # Metrics / traces / logs plugin
|
||
│ ├── image_gen/ # Image-generation providers
|
||
│ └── <others>/ # disk-cleanup, google_meet, platforms, spotify,
|
||
│ # strike-freedom-cockpit, ...
|
||
├── optional-skills/ # Heavier/niche skills shipped but NOT active by default
|
||
├── skills/ # Built-in skills bundled with the repo
|
||
├── ui-tui/ # Ink (React) terminal UI — `hermes --tui`
|
||
│ └── src/ # entry.tsx, app.tsx, gatewayClient.ts + app/components/hooks/lib
|
||
├── tui_gateway/ # Python JSON-RPC backend for the TUI
|
||
├── acp_adapter/ # ACP server (VS Code / Zed / JetBrains integration)
|
||
├── cron/ # Scheduler — jobs.py, scheduler.py
|
||
├── scripts/ # run_tests.sh, release.py, auxiliary scripts
|
||
├── website/ # Docusaurus docs site
|
||
└── tests/ # Pytest suite (~17k tests across ~900 files as of May 2026)
|
||
```
|
||
|
||
**User config:** `~/.hermes/config.yaml` (settings), `~/.hermes/.env` (API keys only).
|
||
**Logs:** `~/.hermes/logs/` — `agent.log` (INFO+), `errors.log` (WARNING+),
|
||
`gateway.log` when running the gateway. Profile-aware via `get_hermes_home()`.
|
||
Browse with `hermes logs [--follow] [--level ...] [--session ...]`.
|
||
|
||
## TypeScript Style
|
||
|
||
Applies to TypeScript across Hermes: desktop, TUI, website, and future TS packages.
|
||
|
||
- Prefer small nanostores over component state when state is shared, reused, or read by distant UI.
|
||
- Let each feature own its atoms. Chat state belongs near chat, shell state near shell, shared state in `src/store`.
|
||
- Components that render from an atom should use `useStore`. Non-rendering actions should read with `$atom.get()`.
|
||
- Do not pass state through three components when the leaf can subscribe to the atom.
|
||
- Keep persistence beside the atom that owns it.
|
||
- Keep route roots thin. They compose routes and shell; they should not become controllers.
|
||
- No monolithic hooks. A hook should own one narrow job.
|
||
- Prefer colocated action modules over hidden god hooks.
|
||
- If a callback is pure side effect, use the terse void form:
|
||
`onState={st => void setGatewayState(st)}`.
|
||
- Async UI handlers should make intent explicit:
|
||
`onClick={() => void save()}`.
|
||
- Prefer interfaces for public props and shared object shapes. Avoid `type X = { ... }` for object props.
|
||
- Extend React primitives for props: `React.ComponentProps<'button'>`, `React.ComponentProps<typeof Dialog>`, `Omit<...>`, `Pick<...>`.
|
||
- Table-driven beats condition ladders when mapping ids, routes, or views.
|
||
- `src/app` owns routes, pages, and page-specific components.
|
||
- `src/store` owns shared atoms.
|
||
- `src/lib` owns shared pure helpers.
|
||
|
||
## File Dependency Chain
|
||
|
||
```
|
||
tools/registry.py (no deps — imported by all tool files)
|
||
↑
|
||
tools/*.py (each calls registry.register() at import time)
|
||
↑
|
||
model_tools.py (imports tools/registry + triggers tool discovery)
|
||
↑
|
||
run_agent.py, cli.py, batch_runner.py, environments/
|
||
```
|
||
|
||
---
|
||
|
||
## AIAgent Class (run_agent.py)
|
||
|
||
The real `AIAgent.__init__` takes ~60 parameters (credentials, routing, callbacks,
|
||
session context, budget, credential pool, etc.). The signature below is the
|
||
minimum subset you'll usually touch — read `run_agent.py` for the full list.
|
||
|
||
```python
|
||
class AIAgent:
|
||
def __init__(self,
|
||
base_url: str = None,
|
||
api_key: str = None,
|
||
provider: str = None,
|
||
api_mode: str = None, # "chat_completions" | "codex_responses" | ...
|
||
model: str = "", # empty → resolved from config/provider later
|
||
max_iterations: int = 500, # tool-calling iterations (shared with subagents)
|
||
enabled_toolsets: list = None,
|
||
disabled_toolsets: list = None,
|
||
quiet_mode: bool = False,
|
||
save_trajectories: bool = False,
|
||
platform: str = None, # "cli", "telegram", etc.
|
||
session_id: str = None,
|
||
skip_context_files: bool = False,
|
||
skip_memory: bool = False,
|
||
credential_pool=None,
|
||
# ... plus callbacks, thread/user/chat IDs, iteration_budget, fallback_model,
|
||
# checkpoints config, prefill_messages, service_tier, reasoning_config, etc.
|
||
): ...
|
||
|
||
def chat(self, message: str) -> str:
|
||
"""Simple interface — returns final response string."""
|
||
|
||
def run_conversation(self, user_message: str, system_message: str = None,
|
||
conversation_history: list = None, task_id: str = None) -> dict:
|
||
"""Full interface — returns dict with final_response + messages."""
|
||
```
|
||
|
||
### Agent Loop
|
||
|
||
The core loop is inside `run_conversation()` — entirely synchronous, with
|
||
interrupt checks, budget tracking, and a one-turn grace call:
|
||
|
||
```python
|
||
while (api_call_count < self.max_iterations and self.iteration_budget.remaining > 0) \
|
||
or self._budget_grace_call:
|
||
if self._interrupt_requested: break
|
||
response = client.chat.completions.create(model=model, messages=messages, tools=tool_schemas)
|
||
if response.tool_calls:
|
||
for tool_call in response.tool_calls:
|
||
result = handle_function_call(tool_call.name, tool_call.args, task_id)
|
||
messages.append(tool_result_message(result))
|
||
api_call_count += 1
|
||
else:
|
||
return response.content
|
||
```
|
||
|
||
Messages follow OpenAI format: `{"role": "system/user/assistant/tool", ...}`.
|
||
Reasoning content is stored in `assistant_msg["reasoning"]`.
|
||
|
||
---
|
||
|
||
## CLI Architecture (cli.py)
|
||
|
||
- **Rich** for banner/panels, **prompt_toolkit** for input with autocomplete
|
||
- **KawaiiSpinner** (`agent/display.py`) — animated faces during API calls, `┊` activity feed for tool results
|
||
- `load_cli_config()` in cli.py merges hardcoded defaults + user config YAML
|
||
- **Skin engine** (`hermes_cli/skin_engine.py`) — data-driven CLI theming; initialized from `display.skin` config key at startup; skins customize banner colors, spinner faces/verbs/wings, tool prefix, response box, branding text
|
||
- `process_command()` is a method on `HermesCLI` — dispatches on canonical command name resolved via `resolve_command()` from the central registry
|
||
- Skill slash commands: `agent/skill_commands.py` scans `~/.hermes/skills/`, injects as **user message** (not system prompt) to preserve prompt caching
|
||
|
||
### Slash Command Registry (`hermes_cli/commands.py`)
|
||
|
||
All slash commands are defined in a central `COMMAND_REGISTRY` list of `CommandDef` objects. Every downstream consumer derives from this registry automatically:
|
||
|
||
- **CLI** — `process_command()` resolves aliases via `resolve_command()`, dispatches on canonical name
|
||
- **Gateway** — `GATEWAY_KNOWN_COMMANDS` frozenset for hook emission, `resolve_command()` for dispatch
|
||
- **Gateway help** — `gateway_help_lines()` generates `/help` output
|
||
- **Telegram** — `telegram_bot_commands()` generates the BotCommand menu
|
||
- **Slack** — `slack_subcommand_map()` generates `/hermes` subcommand routing
|
||
- **Autocomplete** — `COMMANDS` flat dict feeds `SlashCommandCompleter`
|
||
- **CLI help** — `COMMANDS_BY_CATEGORY` dict feeds `show_help()`
|
||
|
||
### Adding a Slash Command
|
||
|
||
1. Add a `CommandDef` entry to `COMMAND_REGISTRY` in `hermes_cli/commands.py`:
|
||
```python
|
||
CommandDef("mycommand", "Description of what it does", "Session",
|
||
aliases=("mc",), args_hint="[arg]"),
|
||
```
|
||
2. Add handler in `HermesCLI.process_command()` in `cli.py`:
|
||
```python
|
||
elif canonical == "mycommand":
|
||
self._handle_mycommand(cmd_original)
|
||
```
|
||
3. If the command is available in the gateway, add a handler in `gateway/run.py`:
|
||
```python
|
||
if canonical == "mycommand":
|
||
return await self._handle_mycommand(event)
|
||
```
|
||
4. For persistent settings, use `save_config_value()` in `cli.py`
|
||
|
||
**CommandDef fields:**
|
||
- `name` — canonical name without slash (e.g. `"background"`)
|
||
- `description` — human-readable description
|
||
- `category` — one of `"Session"`, `"Configuration"`, `"Tools & Skills"`, `"Info"`, `"Exit"`
|
||
- `aliases` — tuple of alternative names (e.g. `("bg",)`)
|
||
- `args_hint` — argument placeholder shown in help (e.g. `"<prompt>"`, `"[name]"`)
|
||
- `cli_only` — only available in the interactive CLI
|
||
- `gateway_only` — only available in messaging platforms
|
||
- `gateway_config_gate` — config dotpath (e.g. `"display.tool_progress_command"`); when set on a `cli_only` command, the command becomes available in the gateway if the config value is truthy. `GATEWAY_KNOWN_COMMANDS` always includes config-gated commands so the gateway can dispatch them; help/menus only show them when the gate is open.
|
||
|
||
**Adding an alias** requires only adding it to the `aliases` tuple on the existing `CommandDef`. No other file changes needed — dispatch, help text, Telegram menu, Slack mapping, and autocomplete all update automatically.
|
||
|
||
---
|
||
|
||
## TUI Architecture (ui-tui + tui_gateway)
|
||
|
||
The TUI is a full replacement for the classic (prompt_toolkit) CLI, activated via `hermes --tui` or `HERMES_TUI=1`.
|
||
|
||
### Process Model
|
||
|
||
```
|
||
hermes --tui
|
||
└─ Node (Ink) ──stdio JSON-RPC── Python (tui_gateway)
|
||
│ └─ AIAgent + tools + sessions
|
||
└─ renders transcript, composer, prompts, activity
|
||
```
|
||
|
||
TypeScript owns the screen. Python owns sessions, tools, model calls, and slash command logic.
|
||
|
||
### Transport
|
||
|
||
Newline-delimited JSON-RPC over stdio. Requests from Ink, events from Python. See `tui_gateway/server.py` for the full method/event catalog.
|
||
|
||
### Key Surfaces
|
||
|
||
| Surface | Ink component | Gateway method |
|
||
|---------|---------------|----------------|
|
||
| Chat streaming | `app.tsx` + `messageLine.tsx` | `prompt.submit` → `message.delta/complete` |
|
||
| Tool activity | `thinking.tsx` | `tool.start/progress/complete` |
|
||
| Approvals | `prompts.tsx` | `approval.respond` ← `approval.request` |
|
||
| Clarify/sudo/secret | `prompts.tsx`, `maskedPrompt.tsx` | `clarify/sudo/secret.respond` |
|
||
| Session picker | `sessionPicker.tsx` | `session.list/resume` |
|
||
| Slash commands | Local handler + fallthrough | `slash.exec` → `_SlashWorker`, `command.dispatch` |
|
||
| Completions | `useCompletion` hook | `complete.slash`, `complete.path` |
|
||
| Theming | `theme.ts` + `branding.tsx` | `gateway.ready` with skin data |
|
||
|
||
### Slash Command Flow
|
||
|
||
1. Built-in client commands (`/help`, `/quit`, `/clear`, `/resume`, `/copy`, `/paste`, etc.) handled locally in `app.tsx`
|
||
2. Everything else → `slash.exec` (runs in persistent `_SlashWorker` subprocess) → `command.dispatch` fallback
|
||
|
||
### Dev Commands
|
||
|
||
```bash
|
||
cd ui-tui
|
||
npm install # first time
|
||
npm run dev # watch mode (rebuilds hermes-ink + tsx --watch)
|
||
npm start # production
|
||
npm run build # full build (hermes-ink + tsc)
|
||
npm run typecheck # typecheck only (tsc --noEmit)
|
||
npm run lint # eslint
|
||
npm run fmt # prettier
|
||
npm test # vitest
|
||
```
|
||
|
||
### TUI in the Dashboard (`hermes dashboard` → `/chat`)
|
||
|
||
The dashboard embeds the real `hermes --tui` — **not** a rewrite. See `hermes_cli/pty_bridge.py` + the `@app.websocket("/api/pty")` endpoint in `hermes_cli/web_server.py`.
|
||
|
||
- Browser loads `web/src/pages/ChatPage.tsx`, which mounts xterm.js's `Terminal` with the WebGL renderer, `@xterm/addon-fit` for container-driven resize, and `@xterm/addon-unicode11` for modern wide-character widths.
|
||
- `/api/pty?token=…` upgrades to a WebSocket; auth uses the same ephemeral `_SESSION_TOKEN` as REST, via query param (browsers can't set `Authorization` on WS upgrade).
|
||
- The server spawns whatever `hermes --tui` would spawn, through `ptyprocess` (POSIX PTY — WSL works, native Windows does not).
|
||
- Frames: raw PTY bytes each direction; resize via `\x1b[RESIZE:<cols>;<rows>]` intercepted on the server and applied with `TIOCSWINSZ`.
|
||
|
||
**Do not re-implement the primary chat experience in React.** The main transcript, composer/input flow (including slash-command behavior), and PTY-backed terminal belong to the embedded `hermes --tui` — anything new you add to Ink shows up in the dashboard automatically. If you find yourself rebuilding the transcript or composer for the dashboard, stop and extend Ink instead.
|
||
|
||
**Structured React UI around the TUI is allowed when it is not a second chat surface.** Sidebar widgets, inspectors, summaries, status panels, and similar supporting views (e.g. `ChatSidebar`, `ModelPickerDialog`, `ToolCall`) are fine when they complement the embedded TUI rather than replacing the transcript / composer / terminal. Keep their state independent of the PTY child's session and surface their failures non-destructively so the terminal pane keeps working unimpaired.
|
||
|
||
### Electron Desktop Chat App (`apps/desktop/`)
|
||
|
||
A **separate** chat surface from both the classic CLI and the dashboard's embedded TUI. It is an Electron + React + nanostore renderer (`@assistant-ui/react`) that talks to a `tui_gateway` backend over JSON-RPC (`requestGateway(method, params)`). The WebSocket/JSON-RPC transport lives in the framework-agnostic `apps/shared` package (`@hermes/shared` — `JsonRpcGatewayClient` + WS URL helpers), which the web dashboard (`web/`) also consumes; **desktop has no build/runtime dependency on the dashboard frontend** — it spawns a headless `hermes serve` backend server (the same gateway `dashboard` serves, minus the browser UI entirely: `serve` sets `headless_backend=True`, so `cmd_dashboard` skips `_build_web_ui` AND exports `HERMES_SERVE_HEADLESS=1` so `mount_spa()` disables the SPA even if a stray `web_dist/` exists — only the JSON-RPC/WS/API surface is reachable). `dashboard` and `serve` share `cmd_dashboard`/`start_server` but are independent surfaces — neither launches the other. The one exception is a backward-compat *fallback*: `serve` is newer, so the desktop spawn (`electron/backend-command.ts` + `backendSupportsServe()` in `electron/main.ts`) detects whether the resolved runtime registers `serve` and, only when it does not (an older managed install / PATH `hermes` the app hasn't updated yet), rewrites the argv to the legacy `dashboard --no-open`. Without that, a new app against an un-upgraded runtime would crash on an unknown subcommand and brick every mid-upgrade user. It does NOT embed `hermes --tui` — it has its own composer, transcript, and slash-command pipeline. For scoped Desktop architecture, state, resolver, transport, and testing rules, read `apps/desktop/AGENTS.md`.
|
||
|
||
**Slash commands in the desktop app are curated client-side, then dispatched to the backend.** The pipeline:
|
||
|
||
- **Backend already provides everything.** `tui_gateway/server.py` `commands.catalog` (empty-query list) and `complete.slash` (typed-query completions) both include built-in commands, user `quick_commands`, AND skill-derived commands (`scan_skill_commands()` / `get_skill_commands()`). The desktop app does not need a new RPC to see skills.
|
||
- **The renderer curates via `apps/desktop/src/lib/desktop-slash-commands.ts`.** This is the load-bearing file. It holds `DESKTOP_COMMAND_SPECS` (the built-ins and their Desktop surfaces) plus `NO_DESKTOP_SURFACE` block-lists for terminal-only / messaging-only / picker-owned / settings-owned / advanced commands that should NOT clutter the desktop popover.
|
||
- `isDesktopSlashCommand(name)` — gates **execution**. Returns true for built-ins AND for any non-built-in (skill / quick command), so typed extension commands run.
|
||
- `isDesktopSlashSuggestion(name)` — gates **discovery/completion**. Used by BOTH completion paths in `app/chat/composer/hooks/use-slash-completions.ts` (empty-query catalog filter + typed-query `complete.slash` filter) and by `filterDesktopCommandsCatalog`.
|
||
- `isDesktopSlashExtensionCommand(name)` — true when the command is NOT a known Hermes built-in (i.e. a skill or user quick command). Both suggestion and catalog-filter paths allow extensions through so skill commands surface in the palette. (Added when fixing "skill commands missing from the desktop slash palette" — the curated allow-list was silently dropping every skill/quick command from completions even though they executed fine when typed.)
|
||
- **Dispatch** lives in `app/session/hooks/use-prompt-actions/slash.ts` (`runSlash`): built-ins that the desktop owns (`/skin`, `/help`, `/new`, …) are handled locally or via `commands.catalog`; everything else goes to `slash.exec`, falling back to `command.dispatch` (which the gateway resolves into skill / alias / exec directives). A skill command resolves to `{type: "skill", message}` and is submitted as a normal prompt.
|
||
|
||
**Rule:** the desktop slash palette's curation is about hiding noise (terminal-only / messaging-only built-ins), NOT about hiding user-activated extensions. Skill commands and `quick_commands` are extensions the backend surfaces — they belong in completions. If you tighten `desktop-slash-commands.ts`, keep `isDesktopSlashExtensionCommand` flowing into both the suggestion and catalog-filter paths. Tests: from `apps/desktop`, run `npx vitest run src/lib/desktop-slash-commands.test.ts` (workspace dependencies are installed at the repo root).
|
||
|
||
---
|
||
|
||
## Adding New Tools
|
||
|
||
Before adding any tool, settle the footprint question first (see "The
|
||
Footprint Ladder" in the Contribution Rubric): most capabilities should NOT
|
||
be core tools. For custom or local-only tools, do **not** edit Hermes core.
|
||
Use the plugin route instead: create `~/.hermes/plugins/<name>/plugin.yaml`
|
||
and `~/.hermes/plugins/<name>/__init__.py`, then register tools with
|
||
`ctx.register_tool(...)`. Plugin toolsets are discovered automatically and can be
|
||
enabled or disabled without touching `tools/` or `toolsets.py`.
|
||
|
||
Use the built-in route below only when the user is explicitly contributing a new
|
||
core Hermes tool that should ship in the base system.
|
||
|
||
Built-in/core tools require changes in **2 files**:
|
||
|
||
**1. Create `tools/your_tool.py`:**
|
||
```python
|
||
import json, os
|
||
from tools.registry import registry
|
||
|
||
def check_requirements() -> bool:
|
||
return bool(os.getenv("EXAMPLE_API_KEY"))
|
||
|
||
def example_tool(param: str, task_id: str = None) -> str:
|
||
return json.dumps({"success": True, "data": "..."})
|
||
|
||
registry.register(
|
||
name="example_tool",
|
||
toolset="example",
|
||
schema={"name": "example_tool", "description": "...", "parameters": {...}},
|
||
handler=lambda args, **kw: example_tool(param=args.get("param", ""), task_id=kw.get("task_id")),
|
||
check_fn=check_requirements,
|
||
requires_env=["EXAMPLE_API_KEY"],
|
||
)
|
||
```
|
||
|
||
**2. Add to `toolsets.py`** — either `_HERMES_CORE_TOOLS` (all platforms) or a new toolset. **This step is required:** auto-discovery imports the tool and registers its schema, but the tool is only *exposed to an agent* if its name appears in a toolset. `_HERMES_CORE_TOOLS` is not dead code — it's the default bundle every platform's base toolset inherits from.
|
||
|
||
Auto-discovery: any `tools/*.py` file with a top-level `registry.register()` call is imported automatically — no manual import list to maintain. Wiring into a toolset is still a deliberate, manual step.
|
||
|
||
The registry handles schema collection, dispatch, availability checking, and error wrapping. All handlers MUST return a JSON string.
|
||
|
||
**Path references in tool schemas**: If the schema description mentions file paths (e.g. default output directories), use `display_hermes_home()` to make them profile-aware. The schema is generated at import time, which is after `_apply_profile_override()` sets `HERMES_HOME`.
|
||
|
||
**State files**: If a tool stores persistent state (caches, logs, checkpoints), use `get_hermes_home()` for the base directory — never `Path.home() / ".hermes"`. This ensures each profile gets its own state.
|
||
|
||
**Agent-level tools** (todo, memory): intercepted by `run_agent.py` before `handle_function_call()`. See `tools/todo_tool.py` for the pattern.
|
||
|
||
---
|
||
|
||
## Dependency Pinning Policy
|
||
|
||
All dependencies must have upper bounds to limit supply-chain attack surface.
|
||
This policy was established after the litellm compromise (PR #2796, #2810) and
|
||
reinforced after the Mini Shai-Hulud worm campaign (May 2026).
|
||
|
||
| Source type | Treatment | Example |
|
||
|---|---|---|
|
||
| PyPI package | `>=floor,<next_major` | `"httpx>=0.28.1,<1"` |
|
||
| Git URL | Commit SHA | `git+https://...@<40-char-sha>` |
|
||
| GitHub Actions | Commit SHA + comment | `uses: actions/checkout@<sha> # v4` |
|
||
| CI-only pip | `==exact` | `pyyaml==6.0.2` |
|
||
|
||
**When adding a new dependency to `pyproject.toml`:**
|
||
1. Pin to `>=current_version,<next_major` for post-1.0 (e.g. `>=1.5.0,<2`).
|
||
2. For pre-1.0 packages, use `<0.(current_minor + 2)` (e.g. `>=0.29,<0.32`).
|
||
3. Never commit a bare `>=X.Y.Z` without a ceiling — CI and reviewers will reject it.
|
||
4. Run `uv lock` to regenerate `uv.lock` with hashes.
|
||
|
||
Reference: #2810 (bounds pass), #9801 (SHA pinning + audit CI).
|
||
|
||
---
|
||
|
||
## Adding Configuration
|
||
|
||
### config.yaml options:
|
||
1. Add to `DEFAULT_CONFIG` in `hermes_cli/config.py`
|
||
2. Bump `_config_version` (check the current value at the top of `DEFAULT_CONFIG`)
|
||
ONLY if you need to actively migrate/transform existing user config
|
||
(renaming keys, changing structure). Adding a new key to an existing
|
||
section is handled automatically by the deep-merge and does NOT require
|
||
a version bump.
|
||
|
||
### Top-level `config.yaml` sections (non-exhaustive):
|
||
|
||
`model`, `agent`, `terminal`, `compression`, `display`, `stt`, `tts`,
|
||
`memory`, `security`, `delegation`, `smart_model_routing`, `checkpoints`,
|
||
`auxiliary`, `curator`, `skills`, `gateway`, `logging`, `cron`, `profiles`,
|
||
`plugins`, `honcho`.
|
||
|
||
`auxiliary` holds per-task overrides for side-LLM work (curator, vision,
|
||
embedding, title generation, session_search, etc.) — each task can pin
|
||
its own provider/model/base_url/max_tokens/reasoning_effort. See
|
||
`agent/auxiliary_client.py::_resolve_auto` for resolution order.
|
||
|
||
`curator` holds the background skill-maintenance config —
|
||
`enabled`, `interval_hours`, `min_idle_hours`, `stale_after_days`,
|
||
`archive_after_days`, `backup` (nested).
|
||
|
||
### .env variables (SECRETS ONLY — API keys, tokens, passwords):
|
||
1. Add to `OPTIONAL_ENV_VARS` in `hermes_cli/config.py` with metadata:
|
||
```python
|
||
"NEW_API_KEY": {
|
||
"description": "What it's for",
|
||
"prompt": "Display name",
|
||
"url": "https://...",
|
||
"password": True,
|
||
"category": "tool", # provider, tool, messaging, setting
|
||
},
|
||
```
|
||
|
||
Non-secret settings (timeouts, thresholds, feature flags, paths, display
|
||
preferences) belong in `config.yaml`, not `.env`. If internal code needs an
|
||
env var mirror for backward compatibility, bridge it from `config.yaml` to
|
||
the env var in code (see `gateway_timeout`, `terminal.cwd` → `TERMINAL_CWD`).
|
||
|
||
### Config loaders (three paths — know which one you're in):
|
||
|
||
| Loader | Used by | Location |
|
||
|--------|---------|----------|
|
||
| `load_cli_config()` | CLI mode | `cli.py` — merges CLI-specific defaults + user YAML |
|
||
| `load_config()` | `hermes tools`, `hermes setup`, most CLI subcommands | `hermes_cli/config.py` — merges `DEFAULT_CONFIG` + user YAML |
|
||
| Direct YAML load | Gateway runtime | `gateway/run.py` + `gateway/config.py` — reads user YAML raw |
|
||
|
||
If you add a new key and the CLI sees it but the gateway doesn't (or vice
|
||
versa), you're on the wrong loader. Check `DEFAULT_CONFIG` coverage.
|
||
|
||
### Working directory:
|
||
- **CLI** — uses the process's current directory (`os.getcwd()`).
|
||
- **Messaging** — uses `terminal.cwd` from `config.yaml`. The gateway bridges this
|
||
to the `TERMINAL_CWD` env var for child tools. **`MESSAGING_CWD` has been
|
||
removed** — the config loader prints a deprecation warning if it's set in
|
||
`.env`. Same for `TERMINAL_CWD` in `.env`; the canonical setting is
|
||
`terminal.cwd` in `config.yaml`.
|
||
|
||
---
|
||
|
||
## Skin/Theme System
|
||
|
||
The skin engine (`hermes_cli/skin_engine.py`) provides data-driven CLI visual customization. Skins are **pure data** — no code changes needed to add a new skin.
|
||
|
||
### Architecture
|
||
|
||
```
|
||
hermes_cli/skin_engine.py # SkinConfig dataclass, built-in skins, YAML loader
|
||
~/.hermes/skins/*.yaml # User-installed custom skins (drop-in)
|
||
```
|
||
|
||
- `init_skin_from_config()` — called at CLI startup, reads `display.skin` from config
|
||
- `get_active_skin()` — returns cached `SkinConfig` for the current skin
|
||
- `set_active_skin(name)` — switches skin at runtime (used by `/skin` command)
|
||
- `load_skin(name)` — loads from user skins first, then built-ins, then falls back to default
|
||
- Missing skin values inherit from the `default` skin automatically
|
||
|
||
### What skins customize
|
||
|
||
| Element | Skin Key | Used By |
|
||
|---------|----------|---------|
|
||
| Banner panel border | `colors.banner_border` | `banner.py` |
|
||
| Banner panel title | `colors.banner_title` | `banner.py` |
|
||
| Banner section headers | `colors.banner_accent` | `banner.py` |
|
||
| Banner dim text | `colors.banner_dim` | `banner.py` |
|
||
| Banner body text | `colors.banner_text` | `banner.py` |
|
||
| Response box border | `colors.response_border` | `cli.py` |
|
||
| Spinner faces (waiting) | `spinner.waiting_faces` | `display.py` |
|
||
| Spinner faces (thinking) | `spinner.thinking_faces` | `display.py` |
|
||
| Spinner verbs | `spinner.thinking_verbs` | `display.py` |
|
||
| Spinner wings (optional) | `spinner.wings` | `display.py` |
|
||
| Tool output prefix | `tool_prefix` | `display.py` |
|
||
| Per-tool emojis | `tool_emojis` | `display.py` → `get_tool_emoji()` |
|
||
| Agent name | `branding.agent_name` | `banner.py`, `cli.py` |
|
||
| Welcome message | `branding.welcome` | `cli.py` |
|
||
| Response box label | `branding.response_label` | `cli.py` |
|
||
| Prompt symbol | `branding.prompt_symbol` | `cli.py` |
|
||
|
||
### Built-in skins
|
||
|
||
- `default` — Classic Hermes gold/kawaii (the current look)
|
||
- `ares` — Crimson/bronze war-god theme with custom spinner wings
|
||
- `mono` — Clean grayscale monochrome
|
||
- `slate` — Cool blue developer-focused theme
|
||
|
||
### Adding a built-in skin
|
||
|
||
Add to `_BUILTIN_SKINS` dict in `hermes_cli/skin_engine.py`:
|
||
|
||
```python
|
||
"mytheme": {
|
||
"name": "mytheme",
|
||
"description": "Short description",
|
||
"colors": { ... },
|
||
"spinner": { ... },
|
||
"branding": { ... },
|
||
"tool_prefix": "┊",
|
||
},
|
||
```
|
||
|
||
### User skins (YAML)
|
||
|
||
Users create `~/.hermes/skins/<name>.yaml`:
|
||
|
||
```yaml
|
||
name: cyberpunk
|
||
description: Neon-soaked terminal theme
|
||
|
||
colors:
|
||
banner_border: "#FF00FF"
|
||
banner_title: "#00FFFF"
|
||
banner_accent: "#FF1493"
|
||
|
||
spinner:
|
||
thinking_verbs: ["jacking in", "decrypting", "uploading"]
|
||
wings:
|
||
- ["⟨⚡", "⚡⟩"]
|
||
|
||
branding:
|
||
agent_name: "Cyber Agent"
|
||
response_label: " ⚡ Cyber "
|
||
|
||
tool_prefix: "▏"
|
||
```
|
||
|
||
Activate with `/skin cyberpunk` or `display.skin: cyberpunk` in config.yaml.
|
||
|
||
---
|
||
|
||
## Plugins
|
||
|
||
Hermes has two plugin surfaces. Both live under `plugins/` in the repo so
|
||
repo-shipped plugins can be discovered alongside user-installed ones in
|
||
`~/.hermes/plugins/` and pip-installed entry points.
|
||
|
||
### General plugins (`hermes_cli/plugins.py` + `plugins/<name>/`)
|
||
|
||
`PluginManager` discovers plugins from `~/.hermes/plugins/`, `./.hermes/plugins/`,
|
||
and pip entry points. Each plugin exposes a `register(ctx)` function that
|
||
can:
|
||
|
||
- Register Python-callback lifecycle hooks:
|
||
`pre_tool_call`, `post_tool_call`, `pre_llm_call`, `post_llm_call`,
|
||
`on_session_start`, `on_session_end`
|
||
- Register new tools via `ctx.register_tool(...)`
|
||
- Register CLI subcommands via `ctx.register_cli_command(...)` — the
|
||
plugin's argparse tree is wired into `hermes` at startup so
|
||
`hermes <pluginname> <subcmd>` works with no change to `main.py`
|
||
|
||
Hooks are invoked from `model_tools.py` (pre/post tool) and `run_agent.py`
|
||
(lifecycle). **Discovery timing pitfall:** `discover_plugins()` only runs
|
||
as a side effect of importing `model_tools.py`. Code paths that read plugin
|
||
state without importing `model_tools.py` first must call `discover_plugins()`
|
||
explicitly (it's idempotent).
|
||
|
||
#### Native plugin compatibility policy
|
||
|
||
The canonical contract and deprecation policy live in
|
||
`website/docs/developer-guide/plugins/index.md#native-plugin-compatibility-contract`.
|
||
Compatibility is enforced as a behavior contract, not through a monolithic
|
||
`PLUGIN_API_VERSION`, a manifest-wide native `api:` match, or version literals
|
||
on unrelated payloads. Keep documented plugin surfaces additive:
|
||
|
||
- add hook payload data as keyword fields; signature-inspect callbacks so old
|
||
narrow signatures receive only fields they declare, while `**kwargs`
|
||
callbacks receive the complete payload;
|
||
- do not remove or rename `PluginContext` methods; make new parameters optional
|
||
with defaults and keyword-only where possible;
|
||
- ignore unknown native manifest fields;
|
||
- give new provider methods default implementations, and signature-inspect
|
||
optional callback kwargs rather than forwarding them unconditionally;
|
||
- use a local schema version only for a capability with a wire or persisted
|
||
contract, and preserve old state/config/session replay or ship a migration.
|
||
|
||
Deprecations require a once-per-process warning, a documented replacement and
|
||
migration note, and at least two subsequent minor releases before removal.
|
||
Compatibility tests must load frozen plugins through the real discovery path
|
||
and assert outcomes. Do not replace these with exact registry/catalog counts,
|
||
source-reading tests, or assertions that a global version literal changed.
|
||
|
||
### Memory-provider plugins (`plugins/memory/<name>/`)
|
||
|
||
Separate discovery system for pluggable memory backends. Current built-in
|
||
providers include **honcho, mem0, supermemory, byterover, hindsight,
|
||
holographic, openviking, retaindb**.
|
||
|
||
Discovery covers the same four sources as the general `PluginManager` —
|
||
bundled, `$HERMES_HOME/plugins/`, `./.hermes/plugins/` (opt-in via
|
||
`HERMES_ENABLE_PROJECT_PLUGINS`), and `hermes_agent.memory_providers` entry
|
||
points — but with **bundled-first** precedence, the reverse of the general
|
||
system's later-wins order: a memory provider is activated by name, so a
|
||
dropped-in directory must not be able to shadow a shipped one. Discovery
|
||
enumerates without importing; nothing runs until `memory.provider` names it.
|
||
|
||
Each provider implements the `MemoryProvider` ABC (see `agent/memory_provider.py`)
|
||
and is orchestrated by `agent/memory_manager.py`. Lifecycle hooks include
|
||
`sync_turn(turn_messages)`, `prefetch(query)`, `shutdown()`, and optional
|
||
`post_setup(hermes_home, config)` for setup-wizard integration.
|
||
|
||
**CLI commands via `plugins/memory/<name>/cli.py`:** if a memory plugin
|
||
defines `register_cli(subparser)`, `discover_plugin_cli_commands()` finds
|
||
it at argparse setup time and wires it into `hermes <plugin>`. The
|
||
framework only exposes CLI commands for the **currently active** memory
|
||
provider (read from `memory.provider` in config.yaml), so disabled
|
||
providers don't clutter `hermes --help`.
|
||
|
||
**Rule (Teknium, May 2026):** plugins MUST NOT modify core files
|
||
(`run_agent.py`, `cli.py`, `gateway/run.py`, `hermes_cli/main.py`, etc.).
|
||
If a plugin needs a capability the framework doesn't expose, expand the
|
||
generic plugin surface (new hook, new ctx method) — never hardcode
|
||
plugin-specific logic into core. PR #5295 removed 95 lines of hardcoded
|
||
honcho argparse from `main.py` for exactly this reason.
|
||
|
||
**No new in-tree memory providers (policy, May 2026):** the set of
|
||
built-in memory providers under `plugins/memory/` is closed. New memory
|
||
backends must ship as **standalone plugin repos** that users install
|
||
into `~/.hermes/plugins/` (or via pip entry points) — they implement
|
||
the same `MemoryProvider` ABC, register through the same discovery
|
||
path, and integrate via `hermes memory setup` / `post_setup()` without
|
||
landing in this tree. PRs that add a new directory under
|
||
`plugins/memory/` will be closed with a pointer to publish the
|
||
provider as its own repo. Existing in-tree providers stay; bug fixes
|
||
to them are welcome.
|
||
|
||
**No new third-party-product plugins in-tree (policy, June 2026):** the
|
||
same rule applies beyond memory providers. Plugins that integrate
|
||
someone else's product or project — observability/metrics backends,
|
||
vendor SaaS connectors, analytics dashboards, paid-service tie-ins —
|
||
must ship as **standalone plugin repos** that users install into
|
||
`~/.hermes/plugins/` (or via pip entry points). They register through
|
||
the existing plugin discovery path and use the ABCs/hooks/ctx surface
|
||
we expose; nothing special is needed in core. The reason is
|
||
maintenance load: every product we absorb into the tree becomes our
|
||
burden to keep working against a fast-moving core, for a backend we
|
||
don't own. Promote standalone plugins in the Nous Research Discord
|
||
(`#plugins-skills-and-skins`). PRs that add such a directory under
|
||
`plugins/` are closed with a pointer to publish it as its own repo —
|
||
this is a coupling decision, not a quality judgment. (The
|
||
`observability/`, `kanban/`, `disk-cleanup/`, etc. directories already
|
||
in the tree are existing precedent, not an invitation to add more
|
||
third-party-product plugins alongside them.)
|
||
|
||
### Model-provider plugins (`plugins/model-providers/<name>/`)
|
||
|
||
Every inference backend (openrouter, anthropic, gmi, deepseek, nvidia, …)
|
||
ships as a plugin here. Each plugin's `__init__.py` calls
|
||
`providers.register_provider(ProviderProfile(...))` at module load.
|
||
`providers/__init__.py._discover_providers()` is a **lazy, separate
|
||
discovery system** — scanned on first `get_provider_profile()` or
|
||
`list_providers()` call, NOT by the general PluginManager.
|
||
|
||
Scan order:
|
||
1. Bundled: `<repo>/plugins/model-providers/<name>/`
|
||
2. User: `$HERMES_HOME/plugins/model-providers/<name>/`
|
||
3. Legacy: `<repo>/providers/<name>.py` (back-compat)
|
||
|
||
User plugins of the same name override bundled ones — `register_provider()`
|
||
is last-writer-wins. This lets third parties swap out any built-in
|
||
profile without a repo patch.
|
||
|
||
The general PluginManager records `kind: model-provider` manifests but does
|
||
NOT import them (would double-instantiate `ProviderProfile`). Plugins
|
||
without an explicit `kind:` get auto-coerced via a source-text heuristic
|
||
(`register_provider` + `ProviderProfile` in `__init__.py`).
|
||
|
||
Full authoring guide: `website/docs/developer-guide/model-provider-plugin.md`.
|
||
|
||
### Dashboard / context-engine / image-gen plugin directories
|
||
|
||
`plugins/context_engine/`, `plugins/image_gen/`, etc. follow the same
|
||
pattern (ABC + orchestrator + per-plugin directory). Context engines
|
||
plug into `agent/context_engine.py`; image-gen providers into
|
||
`agent/image_gen_provider.py`. Reference / docs-companion plugins
|
||
(`example-dashboard`, `strike-freedom-cockpit`, `plugin-llm-example`,
|
||
`plugin-llm-async-example`) live in the
|
||
[`hermes-example-plugins`](https://github.com/NousResearch/hermes-example-plugins)
|
||
companion repo, not in this tree.
|
||
|
||
---
|
||
|
||
## Skills
|
||
|
||
Two parallel surfaces:
|
||
|
||
- **`skills/`** — built-in skills shipped and loadable by default.
|
||
Organized by category directories (e.g. `skills/github/`, `skills/mlops/`).
|
||
- **`optional-skills/`** — heavier or niche skills shipped with the repo but
|
||
NOT active by default. Installed explicitly via
|
||
`hermes skills install official/<category>/<skill>`. Adapter lives in
|
||
`tools/skills_hub.py` (`OptionalSkillSource`). Categories include
|
||
`autonomous-ai-agents`, `blockchain`, `communication`, `creative`,
|
||
`devops`, `email`, `health`, `mcp`, `migration`, `mlops`, `productivity`,
|
||
`research`, `security`, `web-development`.
|
||
|
||
When reviewing skill PRs, check which directory they target — heavy-dep or
|
||
niche skills belong in `optional-skills/`.
|
||
|
||
### SKILL.md frontmatter
|
||
|
||
Standard fields: `name`, `description`, `version`, `author`, `license`,
|
||
`platforms` (OS-gating list: `[macos]`, `[linux, macos]`, ...),
|
||
`metadata.hermes.tags`, `metadata.hermes.category`,
|
||
`metadata.hermes.related_skills`, `metadata.hermes.config` (config.yaml
|
||
settings the skill needs — stored under `skills.config.<key>`, prompted
|
||
during setup, injected at load time).
|
||
|
||
Top-level `tags:` and `category:` are also accepted and mirrored from
|
||
`metadata.hermes.*` by the loader.
|
||
|
||
### Skill authoring standards (HARDLINE)
|
||
|
||
Every new or modernized skill — bundled, optional, or contributed —
|
||
must meet these standards before merge. Reviewers reject PRs that
|
||
violate them.
|
||
|
||
1. **`description` ≤ 60 characters, one sentence, ends with a period.**
|
||
Long descriptions bloat skill listings and dilute the model's
|
||
attention when many skills are loaded. State the capability, not
|
||
the implementation. No marketing words ("powerful",
|
||
"comprehensive", "seamless", "advanced"). Don't repeat the skill
|
||
name. Verify with:
|
||
```python
|
||
import re, pathlib
|
||
m = re.search(r'^description: (.*)$',
|
||
pathlib.Path('skills/<cat>/<name>/SKILL.md').read_text(),
|
||
re.MULTILINE)
|
||
assert len(m.group(1)) <= 60, len(m.group(1))
|
||
```
|
||
|
||
2. **Tools referenced in SKILL.md prose must be native Hermes tools or
|
||
MCP servers the skill explicitly expects.** When the skill needs a
|
||
capability, point at the proper tool by name in backticks
|
||
(`` `terminal` ``, `` `web_extract` ``, `` `read_file` ``,
|
||
`` `patch` ``, `` `search_files` ``, `` `vision_analyze` ``,
|
||
`` `browser_navigate` ``, `` `delegate_task` ``, etc.). Do NOT
|
||
name shell utilities the agent already has wrapped — `grep` →
|
||
`search_files`, `cat`/`head`/`tail` → `read_file`, `sed`/`awk` →
|
||
`patch`, `find`/`ls` → `search_files target='files'`. If the skill
|
||
depends on an MCP server, name the MCP server and document the
|
||
expected setup in `## Prerequisites`. Anything else (third-party
|
||
CLIs, shell pipelines, etc.) is fair game inside script files but
|
||
should not be the headline interaction surface in the prose.
|
||
|
||
3. **`platforms:` gating audited against actual script imports.**
|
||
Skills that use POSIX-only primitives (`fcntl`, `termios`,
|
||
`os.setsid`, `os.kill(pid, 0)` for liveness, `/proc`, `/tmp`
|
||
hardcoded, `signal.SIGKILL`, bash heredocs, `osascript`, `apt`,
|
||
`systemctl`) must declare their supported platforms. Default
|
||
posture: try to fix it cross-platform first — `tempfile.gettempdir`,
|
||
`pathlib.Path`, `psutil.pid_exists`, Python-level filtering instead
|
||
of `grep`. Gate to a narrower set only when the dependency is
|
||
genuinely platform-bound.
|
||
|
||
4. **`author` credits the human contributor first.** For external
|
||
contributions, the contributor's real name + GitHub handle goes
|
||
first; "Hermes Agent" is the secondary collaborator. If the
|
||
contributor's commit shows "Hermes Agent" as author (because they
|
||
used Hermes to draft the skill), replace it with their actual name
|
||
— credit the human, not the tool.
|
||
|
||
5. **SKILL.md body uses the modern section order.** `# <Skill> Skill`
|
||
title, 2-3 sentence intro stating what it does and doesn't do,
|
||
`## When to Use`, `## Prerequisites`, `## How to Run`,
|
||
`## Quick Reference`, `## Procedure`, `## Pitfalls`,
|
||
`## Verification`. Target ~200 lines for a complex skill,
|
||
~100 lines for a simple one. Cut redundant intro fluff, marketing
|
||
prose, and re-explanations of env vars already in
|
||
`## Prerequisites`.
|
||
|
||
6. **Scripts go in `scripts/`, references in `references/`,
|
||
templates in `templates/`.** Don't expect the model to inline-write
|
||
parsers, XML walkers, or non-trivial logic every call — ship a
|
||
helper script. Reference it from SKILL.md by path relative to the
|
||
skill directory.
|
||
|
||
7. **Tests live at `tests/skills/test_<skill>_skill.py`** and use only
|
||
stdlib + pytest + `unittest.mock`. No live network calls. Run via
|
||
`scripts/run_tests.sh tests/skills/test_<skill>_skill.py -q`.
|
||
|
||
8. **`.env.example` additions are isolated to a clearly delimited
|
||
block.** Don't touch the surrounding file — contributor-supplied
|
||
`.env.example` versions are usually stale and edits outside the
|
||
skill's own block must be dropped during salvage.
|
||
|
||
The full salvage / modernization checklist for external skill PRs
|
||
lives in the `hermes-agent-dev` skill at
|
||
`references/new-skill-pr-salvage.md` — load it before polishing
|
||
contributor skill PRs.
|
||
|
||
---
|
||
|
||
## Toolsets
|
||
|
||
All toolsets are defined in `toolsets.py` as a single `TOOLSETS` dict.
|
||
Each platform's adapter picks a base toolset (e.g. Telegram uses
|
||
`"messaging"`); `_HERMES_CORE_TOOLS` is the default bundle most
|
||
platforms inherit from.
|
||
|
||
Current toolset keys: `browser`, `clarify`, `code_execution`, `cronjob`,
|
||
`debugging`, `delegation`, `discord`, `discord_admin`, `feishu_doc`,
|
||
`feishu_drive`, `file`, `homeassistant`, `image_gen`, `kanban`, `memory`,
|
||
`messaging`, `moa`, `rl`, `safe`, `search`, `session_search`, `skills`,
|
||
`spotify`, `terminal`, `todo`, `tts`, `video`, `vision`, `web`, `yuanbao`.
|
||
|
||
Enable/disable per platform via `hermes tools` (the curses UI) or the
|
||
`tools.<platform>.enabled` / `tools.<platform>.disabled` lists in
|
||
`config.yaml`.
|
||
|
||
---
|
||
|
||
## Delegation (`delegate_task`)
|
||
|
||
`tools/delegate_tool.py` spawns a subagent with an isolated
|
||
context + terminal session. By default the parent waits for the
|
||
child's summary before continuing its own loop. With `background=true`,
|
||
Hermes returns a delegation id immediately and the result re-enters the
|
||
conversation later through the async-delegation completion queue.
|
||
|
||
Two shapes:
|
||
|
||
- **Single:** pass `goal` (+ optional `context`, `toolsets`).
|
||
- **Batch (parallel):** pass `tasks: [...]` — each gets its own subagent
|
||
running concurrently. Concurrency is capped by
|
||
`delegation.max_concurrent_children` (default 3).
|
||
|
||
Roles:
|
||
|
||
- `role="leaf"` (default) — focused worker. Cannot call `delegate_task`,
|
||
`clarify`, `memory`, `send_message`, `cronjob`. Retains `execute_code`
|
||
(programmatic tool calling).
|
||
- `role="orchestrator"` — retains `delegate_task` so it can spawn its
|
||
own workers. Gated by `delegation.orchestrator_enabled` (default true)
|
||
and bounded by `delegation.max_spawn_depth` (default 2).
|
||
|
||
Key config knobs (under `delegation:` in `config.yaml`):
|
||
`max_concurrent_children`, `max_spawn_depth`, `child_timeout_seconds`,
|
||
`orchestrator_enabled`, `subagent_auto_approve`, `inherit_mcp_toolsets`,
|
||
`max_iterations`.
|
||
|
||
Durability rule: background `delegate_task` is detached from the current
|
||
turn but still process-local. For work that must survive process restart, use
|
||
`cronjob` or `terminal(background=True, notify_on_complete=True)` instead.
|
||
|
||
---
|
||
|
||
## Curator (skill lifecycle)
|
||
|
||
Background skill-maintenance system that tracks usage on agent-created
|
||
skills and auto-archives stale ones. Users never lose skills; archives
|
||
go to `~/.hermes/skills/.archive/` and are restorable.
|
||
|
||
- **Core:** `agent/curator.py` (review loop, auto-transitions, LLM review
|
||
prompt) + `agent/curator_backup.py` (pre-run tar.gz snapshots).
|
||
- **CLI:** `hermes_cli/curator.py` wires `hermes curator <verb>` where
|
||
verbs are: `status`, `run`, `pause`, `resume`, `pin`, `unpin`,
|
||
`archive`, `restore`, `prune`, `backup`, `rollback`.
|
||
- **Telemetry:** `tools/skill_usage.py` owns the sidecar
|
||
`~/.hermes/skills/.usage.json` — per-skill `use_count`, `view_count`,
|
||
`patch_count`, `last_activity_at`, `state` (active / stale /
|
||
archived), `pinned`.
|
||
|
||
Invariants:
|
||
- Curator only touches skills with `created_by: "agent"` provenance —
|
||
bundled + hub-installed skills are off-limits.
|
||
- Never deletes; max destructive action is archive.
|
||
- Pinned skills are exempt from every auto-transition and from the
|
||
LLM review pass.
|
||
- `skill_manage(action="delete")` refuses pinned skills; patch/edit/
|
||
write_file/remove_file go through so the agent can keep improving
|
||
pinned skills.
|
||
|
||
Config section (`curator:` in `config.yaml`):
|
||
`enabled`, `interval_hours`, `min_idle_hours`, `stale_after_days`,
|
||
`archive_after_days`, `backup.*`.
|
||
|
||
Full user-facing docs: `website/docs/user-guide/features/curator.md`.
|
||
|
||
---
|
||
|
||
## Cron (scheduled jobs)
|
||
|
||
`cron/jobs.py` (job store) + `cron/scheduler.py` (tick loop). Agents
|
||
schedule jobs via the `cronjob` tool; users via `hermes cron <verb>`
|
||
(`list`, `add`, `edit`, `pause`, `resume`, `run`, `remove`) or the
|
||
`/cron` slash command.
|
||
|
||
Supported schedule formats:
|
||
- Duration: `"30m"`, `"2h"`, `"1d"`
|
||
- "every" phrase: `"every 2h"`, `"every monday 9am"`
|
||
- 5-field cron expression: `"0 9 * * *"`
|
||
- ISO timestamp (one-shot): `"2026-06-01T09:00:00Z"`
|
||
|
||
Per-job fields include `skills` (load specific skills), `model` /
|
||
`provider` overrides, `script` (pre-run data-collection script whose
|
||
stdout is injected into the prompt; `no_agent=True` turns the script
|
||
into the entire job), `context_from` (chain job A's last output into
|
||
job B's prompt), `workdir` (run in a specific directory with its
|
||
`AGENTS.md`/`CLAUDE.md` loaded), and multi-platform delivery.
|
||
|
||
Hardening invariants:
|
||
- **3-minute hard interrupt** on cron sessions — runaway agent loops
|
||
cannot monopolize the scheduler.
|
||
- Catchup window: half the job's period, clamped to 120s–2h.
|
||
- Grace window: 120s for one-shot jobs whose fire time was missed.
|
||
- File lock at `~/.hermes/cron/.tick.lock` prevents duplicate ticks
|
||
across processes.
|
||
- Cron sessions pass `skip_memory=True` by default; memory providers
|
||
intentionally do not run during cron.
|
||
|
||
Cron deliveries are **not** mirrored into the target gateway session —
|
||
they land in their own cron session with a header/footer frame so the
|
||
main conversation's message-role alternation stays intact.
|
||
|
||
---
|
||
|
||
## Kanban (multi-agent work queue)
|
||
|
||
Durable SQLite-backed board that lets multiple profiles / workers
|
||
collaborate on shared tasks. Users drive it via `hermes kanban <verb>`;
|
||
workers spawned by the dispatcher drive it via a dedicated `kanban_*`
|
||
toolset so their schema footprint is zero when they're not inside a
|
||
kanban task.
|
||
|
||
- **CLI:** `hermes_cli/kanban.py` wires `hermes kanban` with verbs
|
||
`init`, `create`, `list` (alias `ls`), `show`, `assign`, `link`,
|
||
`unlink`, `comment`, `attach`, `attachments`, `attach-rm`, `complete`,
|
||
`request-review`, `request-changes`, `reopen-review`, `block`, `unblock`, `archive`,
|
||
`tail`, plus less-commonly-used `watch`, `stats`, `runs`, `log`,
|
||
`assignees`, `heartbeat`, `notify-*`, `dispatch`, `daemon`, `gc`.
|
||
- **Worker/orchestrator toolset:** `tools/kanban_tools.py` exposes
|
||
`kanban_show`, `kanban_complete`, `kanban_request_review`,
|
||
`kanban_request_changes`, `kanban_block`,
|
||
`kanban_heartbeat`, `kanban_comment`, `kanban_create`, `kanban_link`,
|
||
`kanban_attach`, `kanban_attach_url`, `kanban_attachments`; profiles that
|
||
explicitly enable the `kanban` toolset outside a dispatcher-spawned
|
||
task also get `kanban_list` and `kanban_unblock` for board routing.
|
||
- **Dispatcher:** long-lived loop that (default every 60s) reclaims
|
||
stale claims, promotes ready tasks, atomically claims, and spawns
|
||
assigned profiles. Runs **inside the gateway** by default via
|
||
`kanban.dispatch_in_gateway: true`.
|
||
- **Plugin assets:** `plugins/kanban/dashboard/` (web UI) +
|
||
`plugins/kanban/systemd/` (`hermes-kanban-dispatcher.service` for
|
||
standalone dispatcher deployment).
|
||
|
||
Isolation model:
|
||
- **Board** is the hard boundary — workers are spawned with
|
||
`HERMES_KANBAN_BOARD` pinned in their env so they can't see other
|
||
boards.
|
||
- **Tenant** is a soft namespace *within* a board — one specialist
|
||
fleet can serve multiple businesses with workspace-path + memory-key
|
||
isolation.
|
||
- After `kanban.failure_limit` consecutive non-success attempts on the
|
||
same task (default: 2), the dispatcher auto-blocks it to prevent spin
|
||
loops.
|
||
|
||
Full user-facing docs: `website/docs/user-guide/features/kanban.md`.
|
||
|
||
---
|
||
|
||
## Important Policies
|
||
|
||
### Prompt Caching Must Not Break
|
||
|
||
Hermes-Agent ensures caching remains valid throughout a conversation. **Do NOT implement changes that would:**
|
||
- Alter past context mid-conversation
|
||
- Change toolsets mid-conversation
|
||
- Reload memories or rebuild system prompts mid-conversation
|
||
|
||
Cache-breaking forces dramatically higher costs. The ONLY time we alter context is during context compression.
|
||
|
||
Slash commands that mutate system-prompt state (skills, tools, memory, etc.)
|
||
must be **cache-aware**: default to deferred invalidation (change takes
|
||
effect next session), with an opt-in `--now` flag for immediate
|
||
invalidation. See `/skills install --now` for the canonical pattern.
|
||
|
||
### Background Process Notifications (Gateway)
|
||
|
||
When `terminal(background=true, notify_on_complete=true)` is used, the gateway runs a watcher that
|
||
detects process completion and triggers a new agent turn. Control verbosity of background process
|
||
messages with `display.background_process_notifications`
|
||
in config.yaml (or `HERMES_BACKGROUND_NOTIFICATIONS` env var):
|
||
|
||
- `concise` — one-line status message on completion; failures append a short output tail (default)
|
||
- `all` — running-output updates + final raw-output message
|
||
- `result` — only the final raw-output completion message
|
||
- `error` — only the final raw-output message when exit code != 0
|
||
- `off` — no watcher messages at all
|
||
|
||
---
|
||
|
||
## Profiles: Multi-Instance Support
|
||
|
||
Hermes supports **profiles** — multiple fully isolated instances, each with its own
|
||
`HERMES_HOME` directory (config, API keys, memory, sessions, skills, gateway, etc.).
|
||
|
||
The core mechanism: `_apply_profile_override()` in `hermes_cli/main.py` sets
|
||
`HERMES_HOME` before any module imports. All `get_hermes_home()` references
|
||
automatically scope to the active profile.
|
||
|
||
### Rules for profile-safe code
|
||
|
||
1. **Use `get_hermes_home()` for all HERMES_HOME paths.** Import from `hermes_constants`.
|
||
NEVER hardcode `~/.hermes` or `Path.home() / ".hermes"` in code that reads/writes state.
|
||
```python
|
||
# GOOD
|
||
from hermes_constants import get_hermes_home
|
||
config_path = get_hermes_home() / "config.yaml"
|
||
|
||
# BAD — breaks profiles
|
||
config_path = Path.home() / ".hermes" / "config.yaml"
|
||
```
|
||
|
||
2. **Use `display_hermes_home()` for user-facing messages.** Import from `hermes_constants`.
|
||
This returns `~/.hermes` for default or `~/.hermes/profiles/<name>` for profiles.
|
||
```python
|
||
# GOOD
|
||
from hermes_constants import display_hermes_home
|
||
print(f"Config saved to {display_hermes_home()}/config.yaml")
|
||
|
||
# BAD — shows wrong path for profiles
|
||
print("Config saved to ~/.hermes/config.yaml")
|
||
```
|
||
|
||
3. **Module-level constants are fine** — they cache `get_hermes_home()` at import time,
|
||
which is AFTER `_apply_profile_override()` sets the env var. Just use `get_hermes_home()`,
|
||
not `Path.home() / ".hermes"`.
|
||
|
||
4. **Tests that mock `Path.home()` must also set `HERMES_HOME`** — since code now uses
|
||
`get_hermes_home()` (reads env var), not `Path.home() / ".hermes"`:
|
||
```python
|
||
with patch.object(Path, "home", return_value=tmp_path), \
|
||
patch.dict(os.environ, {"HERMES_HOME": str(tmp_path / ".hermes")}):
|
||
...
|
||
```
|
||
|
||
5. **Gateway platform adapters should use token locks** — if the adapter connects with
|
||
a unique credential (bot token, API key), call `acquire_scoped_lock()` from
|
||
`gateway.status` in the `connect()`/`start()` method and `release_scoped_lock()` in
|
||
`disconnect()`/`stop()`. This prevents two profiles from using the same credential.
|
||
See `plugins/platforms/irc/adapter.py` for the canonical pattern.
|
||
|
||
6. **Profile operations are HOME-anchored, not HERMES_HOME-anchored** — `_get_profiles_root()`
|
||
returns `Path.home() / ".hermes" / "profiles"`, NOT `get_hermes_home() / "profiles"`.
|
||
This is intentional — it lets `hermes -p coder profile list` see all profiles regardless
|
||
of which one is active.
|
||
|
||
7. **Multiplex profile-scoped env reads MUST fail closed — never borrow from `os.environ`**
|
||
(`agent/secret_scope.py` contract; #72348, #86905). Under `gateway.multiplex_profiles`,
|
||
`os.environ` holds the **default profile's** values; a secondary profile's `.env` lives
|
||
only in its secret scope (installed per-turn by `_profile_runtime_scope`). Any
|
||
profile-level env config — credentials (`app_secret`, tokens) AND authorization
|
||
(`FEISHU_ALLOWED_USERS`, `{PLATFORM}_ALLOW_ALL_USERS`, `GATEWAY_ALLOW_ALL_USERS`,
|
||
`group_policy`, `allow_bots`, ...) — must be read scope-aware:
|
||
- Adapters: `_get_scoped_secret()` (canonical fail-closed copy in
|
||
`plugins/platforms/feishu/adapter.py`, #86905).
|
||
- Gateway authz: `_auth_env()` / `_platform_gate_env()` (`gateway/authz_mixin.py`).
|
||
Rules:
|
||
- Scope installed + multiplex active → a scoped miss returns the **default**.
|
||
NEVER fall through to `os.environ` — that leaks another profile's value and
|
||
silently breaks routing/admission (a leaked default allowlist skips the
|
||
allow-all check and rejects every secondary-profile sender, #86905).
|
||
- Unscoped default-profile path (`UnscopedSecretError`) and single-profile
|
||
deployments keep the `os.environ` read — there it IS the profile's own value.
|
||
- Authorization config is the sharpest edge: allowlist/allow-all leaks cause
|
||
silent rejections (or worse, fail-open) that only show up as missing replies.
|
||
- The `_get_scoped_secret` wrapper is copy-pasted across ~15 platform adapters —
|
||
when touching any of them, make sure the fail-closed semantics are present;
|
||
do not reintroduce the `except _UnscopedSecretError: val = os.getenv(...)`
|
||
fallback-after-miss shape.
|
||
|
||
## Known Pitfalls
|
||
|
||
### DO NOT hardcode `~/.hermes` paths
|
||
Use `get_hermes_home()` from `hermes_constants` for code paths. Use `display_hermes_home()`
|
||
for user-facing print/log messages. Hardcoding `~/.hermes` breaks profiles — each profile
|
||
has its own `HERMES_HOME` directory. This was the source of 5 bugs fixed in PR #3575.
|
||
|
||
### All CLI menu-pickers MUST use curses.
|
||
Interactive menus must use `hermes_cli/curses_ui.py`. See `hermes_cli/tools_config.py` for an example.
|
||
|
||
### DO NOT use `\033[K` (ANSI erase-to-EOL) in spinner/display code
|
||
Leaks as literal `?[K` text under `prompt_toolkit`'s `patch_stdout`. Use space-padding: `f"\r{line}{' ' * pad}"`.
|
||
|
||
### `_last_resolved_tool_names` is a process-global in `model_tools.py`
|
||
`_run_single_child()` in `delegate_tool.py` saves and restores this global around subagent execution. If you add new code that reads this global, be aware it may be temporarily stale during child agent runs.
|
||
|
||
### DO NOT hardcode cross-tool references in schema descriptions
|
||
Tool schema descriptions must not mention tools from other toolsets by name (e.g., `browser_navigate` saying "prefer web_search"). Those tools may be unavailable (missing API keys, disabled toolset), causing the model to hallucinate calls to non-existent tools. If a cross-reference is needed, add it dynamically in `get_tool_definitions()` in `model_tools.py` — see the `browser_navigate` / `execute_code` post-processing blocks for the pattern.
|
||
|
||
### The gateway has TWO message guards — both must bypass approval/control commands
|
||
When an agent is running, messages pass through two sequential guards:
|
||
(1) **base adapter** (`gateway/platforms/base.py`) queues messages in
|
||
`_pending_messages` when `session_key in self._active_sessions`, and
|
||
(2) **gateway runner** (`gateway/run.py`) intercepts `/stop`, `/new`,
|
||
`/queue`, `/status`, `/approve`, `/deny` before they reach
|
||
`running_agent.interrupt()`. Any new command that must reach the runner
|
||
while the agent is blocked (e.g. approval prompts) MUST bypass BOTH
|
||
guards and be dispatched inline, not via `_process_message_background()`
|
||
(which races session lifecycle).
|
||
|
||
### Streaming delivery contract (stream-is-the-message adapters) — duplicate-final class
|
||
Adapters with `draft_stream_is_message = True` (relay Slack native streaming)
|
||
keep ONE cumulative native stream per turn; the stream IS the final message.
|
||
Four invariants, each learned from a live duplicate-final incident (NS-658
|
||
canary ledger, hermes#85796 / gateway-gateway#210). Violating any of them
|
||
re-creates a duplicate or a frozen stream:
|
||
|
||
1. **Draft frames must be prefix-stable.** The connector computes append-only
|
||
deltas: frame N must be a string prefix of frame N+1. NEVER mutate draft
|
||
frames per-tick — no fence-closing (`ensure_closed_code_fences`), no cursor
|
||
suffix, no segment-state resets at tool boundaries, no mrkdwn conversion.
|
||
Any non-prefix frame triggers a whole-snapshot re-append on the platform
|
||
("stacked copies"). The finalize path may still transform the real final.
|
||
2. **The consumer declares the final; the adapter never guesses.**
|
||
`finish(final_text)` carries the completed `final_response` (verifier
|
||
footer, completion explainer included) as the authoritative finalize
|
||
payload. New post-stream response augmentation MUST ride this payload —
|
||
if it mutates `final_response` after the stream sealed, it re-opens the
|
||
#11 bug (`delivered_final_matches` mismatch → corrective duplicate send).
|
||
3. **Interim sends must carry `_interim_send` metadata.** Any consumer-side
|
||
`adapter.send()` that is NOT the turn-final (commentary, segment-tail
|
||
flushes) must set `metadata["_interim_send"] = True`, or the relay
|
||
adapter's seal-interception will seal the live stream with interim text.
|
||
Seal-interception exists at BOTH egress doors (`send()` AND
|
||
`send_for_platform()`); a new egress door needs the same two checks.
|
||
4. **Reconcile by edit, never by plain send.** Any lane that delivers a final
|
||
beside an already-sealed stream (queued follow-ups, media-accompanied
|
||
finals, future lanes) must first try `edit_message` on the consumer's
|
||
`message_id`; plain `send()` is the fallback only when no editable message
|
||
exists. A sealed native stream is a regular message — `chat.update` on it
|
||
works (live-verified).
|
||
|
||
Contract tests: `tests/gateway/test_stream_final_contract.py` (all four
|
||
invariants, mutation-checked). Slack streaming API ground truth (live-probed,
|
||
also encoded in connector comments/tests): `chat.*Stream` speaks STANDARD
|
||
markdown, not mrkdwn; `stopStream.markdown_text` APPENDS (never replaces);
|
||
`startStream`/`stopStream` are rate-limit Tier 2 (~20/min).
|
||
|
||
Guard style note: check `draft_stream_is_message` with `is True` — MagicMock
|
||
adapters in older tests auto-create truthy attributes.
|
||
|
||
### Squash merges from stale branches silently revert recent fixes
|
||
Before squash-merging a PR, ensure the branch is up to date with `main`
|
||
(`git fetch origin main && git reset --hard origin/main` in the worktree,
|
||
then re-apply the PR's commits). A stale branch's version of an unrelated
|
||
file will silently overwrite recent fixes on main when squashed. Verify
|
||
with `git diff HEAD~1..HEAD` after merging — unexpected deletions are a
|
||
red flag.
|
||
|
||
### Don't wire in dead code without E2E validation
|
||
Unused code that was never shipped was dead for a reason. Before wiring an
|
||
unused module into a live code path, E2E test the real resolution chain
|
||
with actual imports (not mocks) against a temp `HERMES_HOME`.
|
||
|
||
### Tests must not write to `~/.hermes/`
|
||
The `_isolate_hermes_home` autouse fixture in `tests/conftest.py` redirects `HERMES_HOME` to a temp dir. Never hardcode `~/.hermes/` paths in tests.
|
||
|
||
**Profile tests**: When testing profile features, also mock `Path.home()` so that
|
||
`_get_profiles_root()` and `_get_default_hermes_home()` resolve within the temp dir.
|
||
Use the pattern from `tests/hermes_cli/test_profiles.py`:
|
||
```python
|
||
@pytest.fixture
|
||
def profile_env(tmp_path, monkeypatch):
|
||
home = tmp_path / ".hermes"
|
||
home.mkdir()
|
||
monkeypatch.setattr(Path, "home", lambda: tmp_path)
|
||
monkeypatch.setenv("HERMES_HOME", str(home))
|
||
return home
|
||
```
|
||
|
||
---
|
||
|
||
## Testing
|
||
|
||
### Python
|
||
**ALWAYS use `scripts/run_tests.sh`** — do not call `pytest` directly. The script enforces
|
||
hermetic environment parity with CI (unset credential vars, TZ=UTC, LANG=C.UTF-8,
|
||
per-file subprocess isolation via `scripts/run_tests_parallel.py` — no xdist,
|
||
worker count auto-scaled from CPU count). Direct `pytest`
|
||
on a 16+ core developer machine with API keys set diverges from CI in ways
|
||
that have caused multiple "works locally, fails in CI" incidents (and the reverse).
|
||
|
||
```bash
|
||
scripts/run_tests.sh # full suite, CI-parity
|
||
scripts/run_tests.sh tests/gateway/ # one directory
|
||
scripts/run_tests.sh tests/agent/test_foo.py -k test_x # one test (file + -k; the runner is file-granular)
|
||
scripts/run_tests.sh -v --tb=long # pass-through pytest flags
|
||
```
|
||
|
||
**Flake policy:** the runner auto-retries a failing test FILE once in a fresh
|
||
subprocess (`--file-retries`, default 1; `HERMES_TEST_FILE_RETRIES=0` to
|
||
disable). Pass-on-retry counts as green but is printed in a `⚠ FLAKY` summary
|
||
section with both attempts' output. A FLAKY report is a bug to fix, not noise
|
||
to ignore — timing-sensitive tests must not assume a quiet runner (loose
|
||
wall-clock bounds ≥ 2s, event-based sync, no `assert not _wait_until(...)`
|
||
negative-timing races).
|
||
|
||
#### Subprocess-per-test-file isolation
|
||
|
||
Every test file runs in a freshly-spawned Python subprocess via `run_tests_parallel.py`. This means module-level dicts/sets and
|
||
ContextVars from one test file cannot leak into the next.
|
||
|
||
#### Why the wrapper
|
||
|
||
| | Without wrapper | With wrapper |
|
||
| ------------------- | ------------------------------------------- | ----------------------------------------- |
|
||
| Provider API keys | Whatever is in your env (auto-detects pool) | All env vars except a specific few unset. |
|
||
| HOME / `~/.hermes/` | Your real config+auth.json | Temp dir per test |
|
||
| Timezone | Local TZ (PDT etc.) | UTC |
|
||
| Locale | Whatever is set | C.UTF-8 |
|
||
|
||
### Where to place what tests
|
||
|
||
The CI change classifier (`scripts/ci/classify_changes.py`) runs specific jobs based on what files changed. A Python test that asserts
|
||
about the contents of `package.json`, `package-lock.json`, `.ts`/`.tsx`
|
||
source, or any other JS-side artifact will not run on a PR that only touches
|
||
those files. This means a regression can go green on a PR and red on `main` (where the
|
||
classifier fails open and runs everything).
|
||
|
||
Any test that reads or asserts about `package.json`,
|
||
`package-lock.json`, `tsconfig.json`, `.ts`/`.tsx`/`.js`/`.mjs`/`.cjs`
|
||
source files configuration belongs in the JS (vitest) test suite, not in `tests/*.py`.
|
||
|
||
### Don't fake the host OS
|
||
|
||
Hermes supports Linux, macOS and native Windows, and plenty of its behaviour
|
||
genuinely differs per host. Those differences are tested by running on the
|
||
host, not by patching `sys.platform`.
|
||
|
||
```python
|
||
@pytest.mark.linux_only
|
||
@pytest.mark.macos_only
|
||
@pytest.mark.windows_only
|
||
```
|
||
|
||
Things that are host-independent can stay unmarked:
|
||
|
||
- **Pure functions that take a platform as data** —
|
||
`hidden_windows_child_options(opts, is_windows=True)` is input→output, not a
|
||
fake host. (Contrast: setting a module-level `IS_WINDOWS` flag and then
|
||
calling `windows_detach_flags()` *is* a fake.)
|
||
- **Declaration/packaging invariants** — "pyproject declares `tzdata` with a
|
||
`sys_platform == 'win32'` marker" asserts about a file, not about runtime.
|
||
|
||
The line: **if the test needs the interpreter to believe it is on another OS
|
||
in order to pass, it belongs on that OS.**
|
||
When one test body walks several platforms in sequence, split it.
|
||
Keep the host-native arm on the Linux lane and move the other arm into its own marked test.
|
||
|
||
**Use the marker, never a bare `skipif`.** `scripts/ci/list_os_marked_tests.py`
|
||
decides which files the macOS/Windows lanes import by grepping for the marker
|
||
*name*, and the lane then filters with `-m <marker>`. A test gated with
|
||
`@pytest.mark.skipif(sys.platform != "win32")` therefore skips on Linux AND is
|
||
never imported on the Windows lane — it runs on no host at all, silently. The
|
||
same trap catches a file-local alias (`windows_only = pytest.mark.skipif(...)`):
|
||
the grep matches the name, so the file *is* listed, but `-m windows_only`
|
||
deselects every test in it and the lane reports green over zero coverage.
|
||
Equally, don't `pytest.skip()` the non-host rows of a `@parametrize` over
|
||
platforms — split it into one marked test per OS, or only the host's row ever
|
||
executes.
|
||
|
||
### Don't write change-detector tests
|
||
|
||
A test is a **change-detector** if it fails whenever data that is **expected
|
||
to change** gets updated — model catalogs, config version numbers,
|
||
enumeration counts, hardcoded lists of provider models. These tests add no
|
||
behavioral coverage; they just guarantee that routine source updates break
|
||
CI and cost engineering time to "fix."
|
||
|
||
**Do not write:**
|
||
|
||
```python
|
||
# catalog snapshot — breaks every model release
|
||
assert "gemini-2.5-pro" in _PROVIDER_MODELS["gemini"]
|
||
assert "MiniMax-M2.7" in models
|
||
|
||
# config version literal — breaks every schema bump
|
||
assert DEFAULT_CONFIG["_config_version"] == 21
|
||
|
||
# enumeration count — breaks every time a skill/provider is added
|
||
assert len(_PROVIDER_MODELS["huggingface"]) == 8
|
||
```
|
||
|
||
**Do write:**
|
||
|
||
```python
|
||
# behavior: does the catalog plumbing work at all?
|
||
assert "gemini" in _PROVIDER_MODELS
|
||
assert len(_PROVIDER_MODELS["gemini"]) >= 1
|
||
|
||
# behavior: does migration bump the user's version to current latest?
|
||
assert raw["_config_version"] == DEFAULT_CONFIG["_config_version"]
|
||
|
||
# invariant: no plan-only model leaks into the legacy list
|
||
assert not (set(moonshot_models) & coding_plan_only_models)
|
||
|
||
# invariant: every model in the catalog has a context-length entry
|
||
for m in _PROVIDER_MODELS["huggingface"]:
|
||
assert m.lower() in DEFAULT_CONTEXT_LENGTHS_LOWER
|
||
```
|
||
|
||
The rule: if the test reads like a snapshot of current data, delete it. If
|
||
it reads like a contract about how two pieces of data must relate, keep it.
|
||
When a PR adds a new provider/model and you want a test, make the test
|
||
assert the relationship (e.g. "catalog entries all have context lengths"),
|
||
not the specific names.
|
||
|
||
Reviewers should reject new change-detector tests; authors should convert
|
||
them into invariants before re-requesting review.
|
||
|
||
### Never read source code in tests
|
||
|
||
A test that reads a source file's text is testing *the shape of the
|
||
source code*, not its behavior. This is a hard antipattern, banned outright.
|
||
Any test that reads a .py, .ts, .tsx, etc., file is suspect.
|
||
|
||
**Why it's actively harmful, not just weak:**
|
||
|
||
- It passes when the implementation is subtly broken (the regex matches a
|
||
call site that exists but is wired wrong) and fails when a correct
|
||
refactor changes formatting, variable names, or control flow with
|
||
identical runtime behavior. Both directions of failure are wrong.
|
||
- It can't be run against a built/bundled/minified artifact, so it silently
|
||
stops testing anything the moment code moves, gets renamed, or a
|
||
dependency reformats it.
|
||
- It actively blocks refactors: reviewers see "keeps a pattern intact" tests
|
||
fail during pure structural cleanup with no behavior change, and either
|
||
hand-wave the failure (dangerous) or waste time updating regexes that add
|
||
nothing (waste).
|
||
- It gives false confidence. a green suite full of source-regex tests
|
||
looks like coverage but has never once executed the code path it claims
|
||
to guard.
|
||
|
||
**Do not write:**
|
||
|
||
```ts
|
||
const source = fs.readFileSync(path.join(__dirname, 'main.ts'), 'utf8')
|
||
|
||
test('backend spawn hides the Windows console', () => {
|
||
assert.match(source, /spawn\(\s*backend\.command,\s*backend\.args[\s\S]{0,300}hiddenWindowsChildOptions/)
|
||
})
|
||
```
|
||
|
||
**Do write — extract the logic into a small pure/DI-testable function and
|
||
call it for real:**
|
||
|
||
```ts
|
||
// backend-spawn.ts
|
||
export function hiddenWindowsChildOptions(options: SpawnOptionsLike = {}, isWindows = process.platform === 'win32') {
|
||
if (!isWindows || 'windowsHide' in options) return options
|
||
return { ...options, windowsHide: true }
|
||
}
|
||
|
||
// backend-spawn.test.ts
|
||
test('windowsHide defaults to true on Windows, is left alone elsewhere', () => {
|
||
assert.equal(hiddenWindowsChildOptions({}, true).windowsHide, true)
|
||
assert.equal(hiddenWindowsChildOptions({}, false).windowsHide, undefined)
|
||
assert.equal(hiddenWindowsChildOptions({ windowsHide: false }, true).windowsHide, false)
|
||
})
|
||
```
|
||
|
||
If the logic lives inline in a god-file (`main.ts`, `cli.py`,
|
||
`gateway/run.py`) and extracting it feels disruptive: that's the actual
|
||
signal to do the extraction, not to regex around it.
|