Files
hermes-agent/tests/wisdom/test_agent_led.py
T
shannonsands a6ee31f55a feat(wisdom): add Hermes Collective Wisdom Agent V1 (#94266)
* feat(wisdom): add trusted publish and install foundation

* feat(wisdom): add private contribution loop

* feat(wisdom): add managed consumption workflows

* fix(wisdom): close cross-repository safety gaps

* fix(wisdom): align local package and lifecycle policy

* fix(wisdom): require explicit profile setup

* docs(wisdom): repin reconciled gateway head

* fix(wisdom): fence content downloads and approval receipts

* docs(wisdom): record generation-fenced downloads

* docs(wisdom): record unified delivery PR

* fix(ci): stop passing invalid classifier inputs

* docs(wisdom): remove internal requirements ledger

* feat(wisdom): localize dashboard and desktop copy

* feat(wisdom): complete local contribution and consumption UX

* style(wisdom): satisfy desktop lint

* chore(wisdom): refresh requirements pin

* test(dashboard): allow formatted profile copy

* test(wisdom): stabilize desktop interaction coverage

* fix(wisdom): surface dashboard action failures

* fix(wisdom): add repeatable Portal demo login

* feat(wisdom): add actionable skill notifications

* feat(wisdom): add notification install and update actions

* fix(wisdom): make Telegram skill alerts actionable

* fix(wisdom): always refresh demo Agent login

* feat(wisdom): embed Telegram notification actions

* fix(wisdom): preserve Telegram notifications after actions

* fix(wisdom): keep Telegram notification cards readable

* feat(wisdom): add Telegram candidate approval flow

* feat(wisdom): explain Telegram qualification reasons

* fix(wisdom): reconcile cross-surface candidate actions

* feat(telegram): add Collective Wisdom management command

* chore(wisdom): refresh Gateway contract pin

* chore(wisdom): advance Gateway contract pin

* feat(wisdom): align command UX across clients

* feat(slack): add Collective Wisdom management parity

* feat(wisdom): add security and professionalism reviews

* feat(wisdom): add first-time qualification guidance

* feat(wisdom): simplify qualification sharing choices

* feat(skills): add optional editorial metadata

* feat(wisdom): enrich legacy skill presentation

* fix(wisdom): harden review and update boundaries

* fix(wisdom): emit canonical review timestamps

* fix(wisdom): align with merged gateway and main

* wisdom: add agent-led sharing core (policy, evidence, schemas, templates, delivery, weekly job, share/install flows)

- hermes_wisdom/agent_led/: policy resolution (server > local > defaults),
  7-day evidence builder that excludes bundled/hub/managed skills and
  dismissed/handled/recently-suggested content hashes, strict pydantic
  schemas for agent output with repair-or-reject, fixed copy templates
  (Share / Teammate / Published / Update / Mute), idempotent retried
  delivery ledger with stale-action resolution, weekly review job,
  resumable Share and Install flows.
- prompts/: candidate review, recipient recommendation, share packaging.
- tests/wisdom/test_agent_led.py: 30 tests.

* wisdom: agent-led renderers and button action dispatcher

- render.py: Telegram HTML, Slack blocks, Desktop payload; editorial name
  is the emphasized line, product label stays separate.
- actions.py: resolve opaque wa:<action>:<dedup> targets via the delivery
  ledger; Not now -> dismissal, Mute -> fixed options, Share -> resumable
  packaging flow, Install/Update -> plan command. Never publishes/installs.

* wisdom: CLI verbs, agent_led config default, conversational catalog skill

- hermes wisdom browse/review-week/act/share/dismiss/mute (all --json).
- wisdom.agent_led config block, default enabled.
- SKILL.md rewritten so natural-language catalog questions map to the CLI
  verbs, share/install flows and fixed notification templates.

* wisdom: wire agent-led weekly review into gateway tick and Telegram buttons

- gateway housekeeping tick calls maybe_run_weekly_review with a home
  channel sender when a Telegram adapter is available.
- Telegram: wa: callbacks resolved through the ledger (stale-safe), mute
  duration keyboard, send_wisdom_agent_recommendation rich card + fallback.

* fix(wisdom): integrate local mediation and harden model and setup boundaries

* fix(wisdom): honor authoritative recommendation policy and defer on failure

* fix(wisdom): synchronize opaque suppression and recheck delivery preferences

* feat(wisdom): route weekly selection through the session-owned assessment queue

* fix(wisdom): prepare and submit the reviewed generated share package

* feat(wisdom): separate native Share preparation from publication consent

* feat(wisdom): sync native mute choices through a leased preference outbox

* feat(wisdom): bind native mute controls to durable preference choices

* feat(wisdom): add scoped desktop and dashboard notification settings

* fix(wisdom): revalidate feed recommendations before assessment and delivery

* fix(wisdom): persist validated delivery receipts before completing notices

* feat(wisdom): add private notification claim and receipt client

* Persist Wisdom send reservations and recover delivery acknowledgements

* Route legacy Wisdom controls through current native review

* Add typed private Wisdom operation outcome client

* fix(wisdom): make agent-led advice usable in the local demo

* fix(wisdom): keep requested consent outside proactive limits

* fix(wisdom): distinguish unavailable assessments and preserve digest text

* fix(wisdom): assess ongoing usefulness beyond the current task

* fix(wisdom): restore immediate qualification sharing controls

* fix(wisdom): separate qualification review from installation advice

* fix(wisdom): collapse review checklists and simplify sharing copy

* fix(wisdom): show compact sharing progress and publication receipts

* fix(wisdom): require credential prefixes rather than matching skill names

* fix(wisdom): finish package checks before presenting sharing consent

* fix(wisdom): scan local skills before qualification cards

* fix(wisdom): update moderation results on existing sharing cards

* fix(wisdom): keep sharing review accessible from receipt cards

* fix(wisdom): align mediated review cards and collapsible checks

* fix(wisdom): clarify clean security summary wording

* fix(wisdom): normalize consent plans and add explicit recheck

* fix(wisdom): keep install and update receipts concise

* fix(wisdom): collapse assessments and deduplicate operation cards

* fix(wisdom): restore private Portal review from native cards

* fix(wisdom): sync Portal publication to original consent card

* fix(wisdom): show local skill version on sharing cards

* fix(wisdom): skip agent recommendations for self-published versions

* fix(wisdom): simplify candidate notices and local-edit recovery copy

* feat(wisdom): submit locally reviewed packages with one confirmation

* feat(wisdom): expose safe receipt and outcome sync recovery

* wisdom: onboarding notice says detect and share, names the user's own skill

Copy review from the product owner on the first and returning
qualification notices (fixed delivery mode):
- the feature blurb now says the org enabled detection *and sharing*
- both notices say the detected skill is one the user created
- both close with an exclamation mark

Applied identically to hermes_wisdom.notice, the desktop and web i18n
strings, and the tests that assert the sentences.

* wisdom: one opener, no approval line, ask to share after the skill is shown

Product owner review of the candidate card.

- The Hermes written card now opens with the same sentence as the fixed card
  ("Your organisation has enabled Collective Wisdom, a feature designed to
  automatically detect and share useful skills across all team members.")
  instead of its own blurb, so there is one first time message.
- "Nothing is shared without your approval." removed from Telegram, Slack
  and Desktop. The buttons already make the permission explicit.
- "Would you like to share?" no longer appears before the skill is named.
  It is now the last line, after the skill name, description, why suggested
  and the checks, and reads "Would you like to share it?" (matching the
  agent led template wording).

Tests updated for the new order; proposalNotice removed from all desktop locales.

* wisdom: American spelling, organization

Product owner decision: user facing copy uses American spelling.
Changes "Your organisation" to "Your organization" in the chat notice,
the Hermes written card opener, the desktop and web strings, and the
tests that assert them. Identifiers such as nas_organisation:* and the
German and French locales are untouched.

* wisdom: candidate card copy round 4 (owner review)

Apply the product owner's round 4 copy decisions to the Hermes Collective
Wisdom candidate card on Telegram, Slack, Desktop and the shared views:

1. Hermes-written cards are titled "Hermes Collective Wisdom" instead of
   the bare "Collective Wisdom".
2. The "Reusable skill ready to review" line is gone from the candidate
   card (Telegram rich card and plain fallback, legacy agent-led share
   template).
3. The skill name and description are labelled: "Skill name: <name>" and
   "What it does: <description>" (Telegram, Slack, Desktop).
4. "Why suggested:" is now "Why others might benefit:".
5. A passing professionalism review reads "Safe to share at work ✓ (no
   inappropriate content found)" with no per-check bullets and no "Pass";
   a failed review reads "Needs a look before sharing at work (possible
   inappropriate content)" and lists only the checks that flagged
   something. Pending/unavailable wording is unchanged.
6. Telegram button toasts: "Will ask later...", "Preparing more
   details...", "Sharing...".
7. Qualification reasons: "You used this skill consistently across many
   days." and "You've really refined this skill."
8. prompts/wisdom_candidate_review.md asks for a compelling
   editorial_name, a simple one_line_description and a compelling
   why_coworkers_benefit under 300 characters; "Be concise and
   convincing." becomes "Be concise and compelling: the goal is that the
   user wants to share it."

Tests updated for the new strings; review_text() gains direct coverage.

* wisdom: re-apply owner copy after rebase

- Native share cards (advice_view/interaction_view): drop the approval line, ask "Would you like to share it?" as the last line after the checks
- Hermes-written completion card titled "Hermes Collective Wisdom"
- Qualification reasons use the owner wording (consistently across many days / really refined)
- American spelling (organization) in remaining English copy
- Desktop test asserts the current Share button; web test matches the returning notice

* fix(wisdom): pin reconciled Gateway and verify Unicode hash vectors

Pin Gateway 60cd2d6b613ae3cd4a6e65155d1142006d907e78 and byte-identical producer artifacts. Verify every content-order case and package-manifest binding. Validation: 186 focused Python tests, Ruff and contract verifier.

* fix(wisdom): reconcile optional SDK tests and frontend lint

* fix(wisdom): default to agent-written notification summaries

* fix(wisdom): restore deferred install review and browse controls

* feat(wisdom): inspect installed setup with exact package provenance

* feat(wisdom): run native-approved installed setup steps with durable evidence

* fix(wisdom): recover interrupted setup with explicit native consent

* feat(wisdom): hand native installs into guided setup review

* fix(wisdom): continue requested setup with fixed notification copy

* fix(wisdom): preserve setup while waiting for a session model

* fix(wisdom): expose canonical setup review controls on desktop

* fix(wisdom): resume setup after recorded automatic updates

* fix(wisdom): make missing setup prerequisites recheckable

* chore(wisdom): align Agent with verified Gateway contract

* fix(wisdom): stop guessing team slugs in portal links

* fix(wisdom): retire pending advice on account sign-out

* fix(wisdom): cancel advice after terminal account revocation

* fix(wisdom): fence feed responses across account sign-out

* fix(wisdom): checkpoint signed-out feed before reactivation

* fix(wisdom): link proactive advice to scoped notification settings

* fix(wisdom): coalesce queued publication recommendations by version

* fix(wisdom): keep package review navigation local and deferable

* fix(wisdom): reflect installed state in discovery controls

* fix(wisdom): show exact checks before command confirmation

* chore(wisdom): pin bounded analytics privacy contract

* chore(wisdom): pin retired legacy notification contract

* feat(wisdom): review publisher usage with exact sharing copy

* fix(wisdom): align discovery and review check summaries

* fix(wisdom): show expired consent before confirmation

* fix(wisdom): require fresh review for legacy install controls

* fix(wisdom): preserve review expiry across check toggles

* fix(wisdom): retain update policy in native install reviews

* fix(wisdom): surface failed native card edits

* fix(wisdom): persist local command approval reviews

* fix(wisdom): use saved approvals for messaging commands

* test(wisdom): provide scan result in setup handoff fixture

* test(wisdom): exercise Telegram approvals with saved review state

* fix(wisdom): retain suppression policy for offline deferral

* fix(wisdom): reconsider candidates after deferred suppression expires

* fix(wisdom): bind review checks and report verified readiness separately

* fix(wisdom): persist accepted publication intent and recover exact outcomes

* fix(sync): pin UTF-8 tree ordering across writers

* chore(wisdom): pin organisation-scoped Gateway authorization

* fix(wisdom): restrict consent delivery to user-facing sessions

* chore(wisdom): refresh reviewed Gateway contract pin

* fix(wisdom): preserve kept tools in Blank Slate exclusions

* test(auth): reset anonymous fixture with a profile-scoped cache

* fix(wisdom): gate local surfaces and work on current profile entitlement

* fix(wisdom): invalidate quiet tool cache on entitlement changes

* test(wisdom): authorize local consent gateway fixtures

* fix(wisdom): keep entitlement decoding free of native crypto imports

* test(wisdom): provide local entitlement to demo CLI subprocess

* ci: leave upstream workflow unchanged in Wisdom PR

* fix(wisdom): ship package and contracts in Nix wheels

---------

Co-authored-by: hbizi <36184542+hbizi@users.noreply.github.com>
2026-09-11 19:04:06 +10:00

605 lines
28 KiB
Python

"""Tests for the agent-led Collective Wisdom sharing layer."""
from __future__ import annotations
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pytest
from hermes_wisdom.agent_led import templates
from hermes_wisdom.agent_led.evidence import build_evidence, provenance_exclusion
from hermes_wisdom.agent_led.history import SuggestionHistory
from hermes_wisdom.agent_led.install_flow import InstallFlow, detect_prerequisites
from hermes_wisdom.agent_led.notify import (
STALE_ACTION_MESSAGE,
DeliveryLedger,
deliver,
share_candidate_event,
teammate_event,
)
from hermes_wisdom.agent_led.policy import AgentLedPolicy, load_policy
from hermes_wisdom.agent_led.schemas import (
SchemaRejected,
parse_candidate_review,
parse_recipient_recommendation,
parse_share_package,
)
from hermes_wisdom.agent_led.share_flow import ShareFlow, scan_credentials
from hermes_wisdom.qualification import record_successful_use
@pytest.fixture(autouse=True)
def _authorize_qualification_work(monkeypatch):
from hermes_cli.config import save_config
from tests.wisdom.local_auth import authorize_local
authorize_local(monkeypatch, "org-1")
save_config({"wisdom": {"enabled": True, "disclosure_acknowledged_at": "fixture"}})
from hermes_wisdom.store import WisdomStore
NOW = datetime(2026, 3, 10, 12, 0, tzinfo=timezone.utc)
# ---------------------------------------------------------------------------
# fixtures
# ---------------------------------------------------------------------------
def _store(tmp_path: Path, *, signed_in: bool = True) -> WisdomStore:
store = WisdomStore(tmp_path / "state")
store.installation_identity()
if signed_in:
store.verify_installation_identity("org-1")
return store
def _skill(root: Path, name: str, *, env: list[str] | None = None) -> Path:
path = root / name
path.mkdir(parents=True)
env_block = ""
if env:
env_block = "required_environment_variables: [" + ", ".join(env) + "]\n"
(path / "SKILL.md").write_text(
f"---\nname: {name}\ndescription: Use when doing {name} work.\n{env_block}---\n# Body\n",
encoding="utf-8",
)
return path
def _make_eligible(monkeypatch, skills_root: Path) -> None:
monkeypatch.setattr("hermes_wisdom.qualification.get_skills_dir", lambda: skills_root)
monkeypatch.setattr(
"hermes_wisdom.qualification._find_skill_dir", lambda name: skills_root / name
)
monkeypatch.setattr("hermes_wisdom.qualification.is_bundled", lambda _n: False)
monkeypatch.setattr("hermes_wisdom.qualification.is_hub_installed", lambda _n: False)
monkeypatch.setattr("hermes_wisdom.qualification._profile_timezone", lambda: (timezone.utc, "UTC"))
monkeypatch.setattr("hermes_wisdom.qualification.ensure_skill_editorial_metadata", lambda p: {"editorial_name": p.name, "editorial_description": "", "changed": False})
monkeypatch.setattr("hermes_wisdom.agent_led.evidence.get_skills_dir", lambda: skills_root)
monkeypatch.setattr("hermes_wisdom.agent_led.evidence.is_bundled", lambda _n: False)
monkeypatch.setattr("hermes_wisdom.agent_led.evidence.is_hub_installed", lambda _n: False)
def _use(store: WisdomStore, name: str, days_ago: list[int], per_day: int = 1) -> None:
for offset in days_ago:
for _ in range(per_day):
record_successful_use(name, store=store, at=NOW - timedelta(days=offset))
def _rec(name: str, content_hash: str, count: int = 5) -> dict:
return {
"skill_name": name,
"content_hash": content_hash,
"editorial_name": "Release Notes Drafter",
"one_line_description": "Drafts release notes from merged PRs.",
"what_it_does": "Collects merged PRs and drafts notes.",
"how_user_relied_on_it": "weekly release announcements",
"evidence": {"window_days": 7, "invocation_count": count, "days_used": 3, "last_used_at": None, "examples": []},
"why_coworkers_benefit": "every team ships weekly",
"audience": "the platform team",
"portability": [],
"confidence": 0.8,
"allowed_actions": ["share", "review", "not_now"],
}
def _review_json(name: str, content_hash: str, count: int = 5) -> str:
return json.dumps(
{
"schema_version": 1,
"window_days": 7,
"considered": [name],
"recommendations": [_rec(name, content_hash, count)],
"nothing_to_recommend_reason": None,
"requires_confirmation": True,
}
)
# ---------------------------------------------------------------------------
# policy
# ---------------------------------------------------------------------------
def test_policy_defaults_and_local_override(monkeypatch):
monkeypatch.setattr("hermes_wisdom.mediation.delivery_mode", lambda: "agent")
policy = load_policy(local={})
assert policy.enabled is False # no authoritative organization policy
assert policy.window_days == 7
assert policy.min_aggregate_count == 3
assert policy.dismiss_suppression_days == 30
assert policy.popular_install_threshold == 10
custom = load_policy(local={"enabled": False, "window_days": 14, "min_aggregate_count": 0})
assert custom.enabled is False
assert custom.window_days == 14
assert custom.min_aggregate_count == 1 # floor
assert custom.source == "local_config"
def test_policy_server_block_wins(monkeypatch):
from hermes_wisdom.client import AgentLedPolicyResponse
monkeypatch.setattr("hermes_wisdom.mediation.delivery_mode", lambda: "agent")
class Client:
display_org_id = "org-1"
def agent_led_policy(self):
return AgentLedPolicyResponse(
org_id="org-1", usage_evidence_window_days=10,
min_aggregate_invocations=8, consecutive_day_usage_counts=False,
repeated_edits_count=False, max_recommendations_per_user_per_week=0,
publication_mode="moderated", install_popularity_threshold=20,
notification_defaults={"skill_ready_to_share": False, "teammate_published": True, "update_available": False},
manager_review_email_cadence="daily", not_now_suppression_days=60,
version=1, updated_by_user_id=None,
)
policy = load_policy(client=Client(), local={"window_days": 5})
assert policy.window_days == 10
assert policy.source == "server_policy"
assert policy.enabled is True
assert policy.min_aggregate_count == 8
assert policy.max_candidates == 0
assert policy.dismiss_suppression_days == 60
assert policy.notification_defaults["skill_ready_to_share"] is False
assert policy.consecutive_day_usage_counts is False
# ---------------------------------------------------------------------------
# evidence: built-in exclusion + 7-day aggregate counting
# ---------------------------------------------------------------------------
def test_builtin_skills_excluded(monkeypatch, tmp_path):
skills = tmp_path / "skills"
_skill(skills, "own-skill")
_skill(skills, "bundled-skill")
_make_eligible(monkeypatch, skills)
store = _store(tmp_path)
_use(store, "own-skill", [0, 1, 2], per_day=2)
_use(store, "bundled-skill", [0, 1, 2], per_day=2)
monkeypatch.setattr(
"hermes_wisdom.agent_led.evidence.is_bundled", lambda n: n == "bundled-skill"
)
payload = build_evidence(
store=store, policy=AgentLedPolicy(), history=SuggestionHistory(tmp_path / "h.json"), at=NOW, skills_root=skills
)
names = [c.skill_name for c in payload.candidates]
assert names == ["own-skill"]
assert payload.excluded["bundled"] == ["bundled-skill"]
assert provenance_exclusion("x", skills / "_wisdom" / "org" / "x", skills_root=skills) == "managed_or_archived"
def test_seven_day_aggregate_counting(monkeypatch, tmp_path):
skills = tmp_path / "skills"
_skill(skills, "counted", env=["MY_TOKEN"])
_make_eligible(monkeypatch, skills)
store = _store(tmp_path)
# 2 uses/day on 3 distinct days inside the window, plus 4 uses outside it.
_use(store, "counted", [0, 3, 6], per_day=2)
_use(store, "counted", [7, 8], per_day=2)
payload = build_evidence(
store=store, policy=AgentLedPolicy(), history=SuggestionHistory(tmp_path / "h.json"), at=NOW, skills_root=skills
)
assert len(payload.candidates) == 1
candidate = payload.candidates[0]
assert candidate.invocation_count == 6
assert candidate.days_used == 3
assert candidate.last_used_day == NOW.date().isoformat()
assert candidate.required_environment_variables == ["MY_TOKEN"]
assert payload.window_start == (NOW.date() - timedelta(days=6)).isoformat()
def test_below_min_count_excluded(monkeypatch, tmp_path):
skills = tmp_path / "skills"
_skill(skills, "rare")
_make_eligible(monkeypatch, skills)
store = _store(tmp_path)
_use(store, "rare", [0])
payload = build_evidence(
store=store, policy=AgentLedPolicy(min_aggregate_count=3), history=SuggestionHistory(tmp_path / "h.json"), at=NOW, skills_root=skills
)
assert payload.candidates == []
assert payload.excluded["below_min_count"] == ["rare"]
# ---------------------------------------------------------------------------
# dismissal suppression by hash
# ---------------------------------------------------------------------------
def test_dismissal_suppresses_same_hash_only(tmp_path):
history = SuggestionHistory(tmp_path / "h.json")
result = history.record_dismissal("s", "hash-a", suppression_days=30, at=NOW)
assert result["gateway"] == "pending_gateway"
assert history.is_suppressed("s", "hash-a", at=NOW + timedelta(days=29))
assert not history.is_suppressed("s", "hash-a", at=NOW + timedelta(days=31))
assert not history.is_suppressed("s", "hash-b", at=NOW) # content changed
reloaded = SuggestionHistory(tmp_path / "h.json")
assert reloaded.is_suppressed("s", "hash-a", at=NOW)
assert reloaded.pending_gateway()[0]["kind"] == "dismissal"
def test_dismissed_skill_excluded_from_evidence(monkeypatch, tmp_path):
skills = tmp_path / "skills"
_skill(skills, "dismissed")
_make_eligible(monkeypatch, skills)
store = _store(tmp_path)
_use(store, "dismissed", [0, 1, 2], per_day=2)
history = SuggestionHistory(tmp_path / "h.json")
first = build_evidence(store=store, policy=AgentLedPolicy(), history=history, at=NOW, skills_root=skills)
history.record_dismissal("dismissed", first.candidates[0].content_hash, at=NOW)
second = build_evidence(store=store, policy=AgentLedPolicy(), history=history, at=NOW, skills_root=skills)
assert second.candidates == []
assert second.excluded["dismissed"] == ["dismissed"]
def test_mute_uses_fixed_options(tmp_path):
history = SuggestionHistory(tmp_path / "h.json")
assert [label for _k, label, _d in templates.MUTE_OPTIONS] == ["1 day", "1 week", "30 days", "Forever"]
history.record_mute("*", days=templates.mute_duration_days("1w"), at=NOW)
assert history.is_muted(at=NOW + timedelta(days=6))
assert not history.is_muted(at=NOW + timedelta(days=8))
history.record_mute("skill-x", days=templates.mute_duration_days("forever"), at=NOW)
assert history.is_muted("skill-x", at=NOW + timedelta(days=4000))
# ---------------------------------------------------------------------------
# schema validation
def test_credential_detector_allows_long_skill_names():
from hermes_wisdom.agent_led.schemas import PackagedFile
text = "Use skill-qualification-test and skill-publishing-flow-test."
assert PackagedFile(path="SKILL.md", content=text).content == text
@pytest.mark.parametrize("prefix", ["sk-", "ghp_", "gho_", "xoxb-", "AKIA", "AIza"])
def test_credential_detector_still_rejects_key_shapes(prefix):
from hermes_wisdom.agent_led.schemas import PackagedFile
with pytest.raises(ValueError, match="credential-shaped"):
PackagedFile(path="SKILL.md", content=prefix + "a" * 32)
# ---------------------------------------------------------------------------
def test_candidate_review_schema_accepts_valid_and_repairs_bad_item():
good = parse_candidate_review(_review_json("s", "abcdef1234"))
assert good.recommendations[0].skill_name == "s"
assert good.requires_confirmation is True
payload = json.loads(_review_json("s", "abcdef1234"))
payload["recommendations"].append({"skill_name": "broken"}) # invalid item
repaired = parse_candidate_review(json.dumps(payload))
assert [r.skill_name for r in repaired.recommendations] == ["s"]
def test_candidate_review_schema_rejects_secrets_and_garbage():
payload = json.loads(_review_json("s", "abcdef1234"))
payload["recommendations"][0]["what_it_does"] = "uses token ghp_" + "a" * 36
# A credential-shaped recommendation is dropped by the repair pass, never surfaced.
assert parse_candidate_review(json.dumps(payload)).recommendations == []
with pytest.raises(SchemaRejected):
parse_candidate_review(json.dumps({**payload, "window_days": 0}))
with pytest.raises(SchemaRejected):
parse_candidate_review("not json at all")
with pytest.raises(SchemaRejected):
parse_candidate_review(json.dumps({**json.loads(_review_json("s", "abcdef1234")), "requires_confirmation": False}))
def test_candidate_review_never_removes_not_now():
payload = json.loads(_review_json("s", "abcdef1234"))
payload["recommendations"][0]["allowed_actions"] = ["share"]
parsed = parse_candidate_review(json.dumps(payload))
assert "not_now" in parsed.recommendations[0].allowed_actions
def test_recipient_recommendation_schema():
irrelevant = parse_recipient_recommendation(
json.dumps({"schema_version": 1, "skill_id": "sk", "version": 1, "relevant": False, "not_relevant_reason": "no overlap", "confidence": 0.9})
)
assert irrelevant.relevant is False and "mute" in irrelevant.allowed_actions
with pytest.raises(SchemaRejected):
parse_recipient_recommendation(
json.dumps({"schema_version": 1, "skill_id": "sk", "version": 1, "relevant": True, "confidence": 0.9})
)
with pytest.raises(SchemaRejected):
parse_recipient_recommendation(
json.dumps({"schema_version": 1, "skill_id": "sk", "version": 1, "relevant": False, "confidence": 0.9, "installed": True})
)
def test_share_package_schema_requires_skill_md_and_safe_paths():
base = {
"schema_version": 1, "skill_name": "s", "source_content_hash": "abcdef1234", "editorial_name": "S",
"plain_description": "Does S.", "files": [{"path": "SKILL.md", "content": "# S"}],
"requirements": [], "setup_instructions": [], "credential_handoff": [], "compatibility_limits": [],
"verification_step": "hermes skills list | grep S", "removed_or_generalized": [], "related_skills": [],
}
assert parse_share_package(json.dumps(base)).skill_name == "s"
with pytest.raises(SchemaRejected):
parse_share_package(json.dumps({**base, "files": [{"path": "README.md", "content": "x"}]}))
with pytest.raises(SchemaRejected):
parse_share_package(json.dumps({**base, "files": [{"path": "../SKILL.md", "content": "x"}]}))
# ---------------------------------------------------------------------------
# templates: exact headers, no 'Pass'
# ---------------------------------------------------------------------------
def test_share_template_exact_strings():
notice = templates.render_share(
editorial_name="Release Notes Drafter",
description="Drafts release notes from merged PRs.",
count_7d=9,
specific_work="weekly release announcements",
audience="the platform team",
reason="every team ships weekly",
checks_passed=True,
)
text = notice.text
assert notice.title == "Hermes Collective Wisdom"
assert "Reusable skill ready to review" not in text
assert notice.lines[0] == "Release Notes Drafter"
assert notice.lines[1] == " Drafts release notes from merged PRs."
assert "Why we ask" in notice.lines
assert (
"You used this skill 9 times in the last 7 days. It has helped with weekly release "
"announcements. I think it could help the platform team because every team ships weekly."
) in notice.lines
assert "Checks complete: no issues detected." in notice.lines
for check in (
"No profanity or abusive language",
"No hate or harassment",
"No sexual or graphic content",
"No detected credentials or private keys",
):
assert f"✓ {check}" in notice.lines
assert notice.lines[-1] == "Would you like to share it?"
assert [a.label for a in notice.actions] == ["Not now", "Review", "Share"]
assert "Pass" not in text.split()
assert "Pass" not in text
def test_share_template_requires_real_count():
with pytest.raises(ValueError):
templates.render_share(
editorial_name="X", description="d", count_7d=None, # type: ignore[arg-type]
specific_work="w", audience="a", reason="r",
)
def test_teammate_template_exact_strings():
notice = templates.render_teammate(
editorial_name="Incident Timeline Builder",
outcome_one_liner="Turn a noisy incident channel into a clean timeline.",
publisher_name="Dana",
specific_work="post-incident reviews",
count_7d=12,
recipient_reason="your on-call rotation",
installation_count=14,
)
assert notice.title == "New skill from your team"
assert notice.lines[0] == "Incident Timeline Builder"
assert notice.lines[2] == "Published by Dana"
assert (
"Dana uses this for post-incident reviews. It was used 12 times in the last 7 days. "
"I think it could help you with your on-call rotation."
) in notice.lines
assert "✓ Security check complete (credentials, private keys, organization policy)" in notice.lines
assert "Popular with your team: 10+ installations" in notice.lines
assert [a.label for a in notice.actions] == ["Mute", "View", "Install"]
assert "New notification" not in notice.text
assert "Pass" not in notice.text
quiet = templates.render_teammate(
editorial_name="X", outcome_one_liner="o", publisher_name="P", specific_work="w",
count_7d=1, recipient_reason="r", installation_count=3,
)
assert not any(line.startswith("Popular with your team") for line in quiet.lines)
def test_published_and_update_templates():
open_notice = templates.render_published(organization_name="Acme", moderated=False)
assert open_notice.title == "Published to Acme."
assert open_notice.lines == ["Your teammates can now find and install this skill."]
assert [a.label for a in open_notice.actions] == ["View in Portal"]
moderated = templates.render_published(organization_name="Acme", moderated=True)
assert moderated.title == "Sent for review."
assert moderated.lines == [
"It is not available to your organization yet. You will be notified when the review is complete."
]
update = templates.render_update(skill_name="Release Notes Drafter", summary="Adds changelog links.", reason="you ship weekly", setup_impact=None)
assert update.title == "Update available for Release Notes Drafter"
assert update.lines == [
"Adds changelog links.",
"This matters for your work because you ship weekly.",
"Setup impact: None",
]
assert [a.label for a in update.actions] == ["Mute", "View changes", "Update"]
for notice in (open_notice, moderated, update):
assert "Pass" not in notice.text
def test_templates_reject_forbidden_words_in_prose():
with pytest.raises(ValueError):
templates.render_update(skill_name="S", summary="Pass", reason="r")
# ---------------------------------------------------------------------------
# idempotent delivery + stale buttons
# ---------------------------------------------------------------------------
def test_legacy_delivery_cannot_bypass_native_ownership(tmp_path):
from unittest.mock import Mock
from hermes_wisdom.agent_led.notify import DeliveryError
from hermes_wisdom.agent_led.schemas import CandidateRecommendation
rec = CandidateRecommendation.model_validate(_rec("s", "abcdef1234"))
event = share_candidate_event(rec, organization_id="org-1", recipient_id="u1", at=NOW)
sender = Mock()
ledger = DeliveryLedger(tmp_path / "ledger.json")
with pytest.raises(DeliveryError, match="profile-owned mediation"):
deliver(event, sender=sender, ledger=ledger, retries=2, sleep=lambda _s: None, now=NOW)
sender.assert_not_called()
assert ledger.get(event.dedup_key) is None
# Same inputs -> same dedup key across processes.
again = share_candidate_event(rec, organization_id="org-1", recipient_id="u1", at=NOW)
assert again.dedup_key == event.dedup_key
def test_legacy_delivery_does_not_modify_existing_snapshots(tmp_path):
from unittest.mock import Mock
from hermes_wisdom.agent_led.notify import DeliveryError
from hermes_wisdom.agent_led.schemas import CandidateRecommendation
rec = CandidateRecommendation.model_validate(_rec("s", "abcdef1234"))
event = share_candidate_event(rec, organization_id="org-1", recipient_id="u1", at=NOW)
ledger = DeliveryLedger(tmp_path / "ledger.json")
ledger.record(event, state="delivered")
before = ledger.path.read_bytes()
sender = Mock()
with pytest.raises(DeliveryError):
deliver(event, sender=sender, ledger=ledger, now=NOW)
sender.assert_not_called()
assert ledger.path.read_bytes() == before
def test_stale_and_duplicate_actions(tmp_path):
from hermes_wisdom.agent_led.schemas import CandidateRecommendation
rec = CandidateRecommendation.model_validate(_rec("s", "abcdef1234"))
event = share_candidate_event(rec, organization_id="org-1", recipient_id="u1", at=NOW, ttl_hours=1)
ledger = DeliveryLedger(tmp_path / "ledger.json")
ledger.record(event, state="delivered")
share_target = next(a.target for a in event.allowed_actions if a.id == "share")
resolved = ledger.resolve_action(share_target, at=NOW)
assert resolved["ok"] and resolved["action"] == "share"
assert ledger.mark_acted(event.dedup_key, "share") is True
assert ledger.mark_acted(event.dedup_key, "share") is False # no duplicate action
assert ledger.resolve_action(share_target, at=NOW)["message"] == STALE_ACTION_MESSAGE
fresh = share_candidate_event(rec, organization_id="org-2", recipient_id="u1", at=NOW, ttl_hours=1)
ledger.record(fresh, state="delivered")
expired = ledger.resolve_action(fresh.allowed_actions[0].target, at=NOW + timedelta(hours=2))
assert expired["stale"] and expired["message"] == STALE_ACTION_MESSAGE
assert ledger.resolve_action("garbage", at=NOW)["stale"]
def test_teammate_event_requires_relevance():
from hermes_wisdom.agent_led.schemas import RecipientRecommendation
rec = RecipientRecommendation.model_validate(
{"schema_version": 1, "skill_id": "sk", "version": 1, "relevant": False, "confidence": 0.5}
)
with pytest.raises(ValueError):
teammate_event(rec, organization_id="o", recipient_id="r")
# ---------------------------------------------------------------------------
# share flow: no publish without confirmation
# ---------------------------------------------------------------------------
def _package_json(name: str, content_hash: str, *, leak: bool = False) -> str:
body = "# Notes\n" + ("token = ghp_" + "b" * 36 if leak else "Generalized body.")
return json.dumps(
{
"schema_version": 1, "skill_name": name, "source_content_hash": content_hash,
"editorial_name": "Release Notes", "plain_description": "Drafts release notes.",
"files": [{"path": "SKILL.md", "content": body, "generalized_from_original": True}],
"requirements": [{"kind": "command", "name": "gh", "purpose": "read merged PRs", "handoff": "install GitHub CLI"}],
"setup_instructions": ["Install gh", "Authenticate gh"],
"credential_handoff": ["A GitHub token with repo read scope, from your own account"],
"compatibility_limits": ["Requires GitHub-hosted repositories"],
"verification_step": "gh --version",
"removed_or_generalized": ["Replaced a home directory path with $HOME"],
"related_skills": [],
}
)
def test_share_flow_never_publishes_without_approval(tmp_path):
skill = _skill(tmp_path / "skills", "notes")
(skill / "SKILL.md").write_text(
(skill / "SKILL.md").read_text() + "\nSee /home/alice/notes and token = ghp_" + "c" * 36 + "\n"
)
flow = ShareFlow(root=tmp_path / "flows")
summary = flow.start(skill)
assert summary["status"] == "prepass"
assert any("Credential-shaped" in p for p in summary["portability_problems"])
assert any("Organization-specific" in p for p in summary["portability_problems"])
published = {"n": 0}
def submit(package, _state):
published["n"] += 1
return {"draft_id": "d1", "published": False}
with pytest.raises(RuntimeError):
flow.approve(submit=submit) # not packaged yet
assert published["n"] == 0
content_hash = summary["source_content_hash"]
flow.package(model_call=lambda m, s: _package_json("notes", content_hash))
assert flow.status == "awaiting_approval"
assert flow.summary()["next_actions"] == ["approve", "request_changes", "cancel"]
assert published["n"] == 0 # packaging alone publishes nothing
resumed = ShareFlow(flow.flow_id, root=tmp_path / "flows")
assert resumed.status == "awaiting_approval"
resumed.request_changes("mention rate limits")
assert resumed.status == "changes_requested"
resumed.package(model_call=lambda m, s: _package_json("notes", content_hash))
resumed.approve(submit=submit)
assert published["n"] == 1 and resumed.status == "submitted"
assert resumed.summary()["published"] is False # draft only; owner review still required
def test_share_flow_rejects_leaked_credentials_in_package(tmp_path):
skill = _skill(tmp_path / "skills", "notes")
flow = ShareFlow(root=tmp_path / "flows")
content_hash = flow.start(skill)["source_content_hash"]
with pytest.raises(SchemaRejected):
flow.package(model_call=lambda m, s: _package_json("notes", content_hash, leak=True))
assert flow.status == "prepass"
assert scan_credentials([{"path": "a", "content": "AKIA" + "A" * 16}])[0]["kind"] == "aws_access_key"
# ---------------------------------------------------------------------------
# install flow: success only after verification, resumable
# ---------------------------------------------------------------------------
def test_install_flow_reports_installed_only_after_verification(tmp_path):
package = json.loads(_package_json("notes", "abcdef1234"))
package["requirements"].append({"kind": "env_var", "name": "NOTES_TOKEN", "purpose": "auth"})
flow = InstallFlow(root=tmp_path / "installs")
flow.start(skill_id="sk-1", version=2, package=package)
summary = flow.check_prerequisites(env={})
assert summary["step"] == "setup" and any(m["name"] == "NOTES_TOKEN" for m in summary["missing"])
assert summary["installed"] is False
resumed = InstallFlow(flow.flow_id, root=tmp_path / "installs")
assert resumed.step == "setup"
resumed.mark_setup_complete("NOTES_TOKEN")
for item in resumed.state["prerequisites"]:
item["status"] = "present"
resumed.state["step"] = "apply"
resumed.apply(lambda _state: {"installed": True, "managed_path": "x"})
assert resumed.step == "verify" and resumed.installed is False
failed = resumed.verify(runner=lambda cmd: (False, "boom"))
assert failed["installed"] is False and failed["files_installed"] is True
assert failed["message"] == "Files installed; setup verification is incomplete."
passed = resumed.verify(runner=lambda cmd: (True, "ok"))
assert passed["installed"] is True and passed["message"] == "Installed and verified."
assert detect_prerequisites([{"kind": "command", "name": "definitely-not-a-real-binary-xyz"}])[0]["status"] == "missing"