a6ee31f55a
* feat(wisdom): add trusted publish and install foundation
* feat(wisdom): add private contribution loop
* feat(wisdom): add managed consumption workflows
* fix(wisdom): close cross-repository safety gaps
* fix(wisdom): align local package and lifecycle policy
* fix(wisdom): require explicit profile setup
* docs(wisdom): repin reconciled gateway head
* fix(wisdom): fence content downloads and approval receipts
* docs(wisdom): record generation-fenced downloads
* docs(wisdom): record unified delivery PR
* fix(ci): stop passing invalid classifier inputs
* docs(wisdom): remove internal requirements ledger
* feat(wisdom): localize dashboard and desktop copy
* feat(wisdom): complete local contribution and consumption UX
* style(wisdom): satisfy desktop lint
* chore(wisdom): refresh requirements pin
* test(dashboard): allow formatted profile copy
* test(wisdom): stabilize desktop interaction coverage
* fix(wisdom): surface dashboard action failures
* fix(wisdom): add repeatable Portal demo login
* feat(wisdom): add actionable skill notifications
* feat(wisdom): add notification install and update actions
* fix(wisdom): make Telegram skill alerts actionable
* fix(wisdom): always refresh demo Agent login
* feat(wisdom): embed Telegram notification actions
* fix(wisdom): preserve Telegram notifications after actions
* fix(wisdom): keep Telegram notification cards readable
* feat(wisdom): add Telegram candidate approval flow
* feat(wisdom): explain Telegram qualification reasons
* fix(wisdom): reconcile cross-surface candidate actions
* feat(telegram): add Collective Wisdom management command
* chore(wisdom): refresh Gateway contract pin
* chore(wisdom): advance Gateway contract pin
* feat(wisdom): align command UX across clients
* feat(slack): add Collective Wisdom management parity
* feat(wisdom): add security and professionalism reviews
* feat(wisdom): add first-time qualification guidance
* feat(wisdom): simplify qualification sharing choices
* feat(skills): add optional editorial metadata
* feat(wisdom): enrich legacy skill presentation
* fix(wisdom): harden review and update boundaries
* fix(wisdom): emit canonical review timestamps
* fix(wisdom): align with merged gateway and main
* wisdom: add agent-led sharing core (policy, evidence, schemas, templates, delivery, weekly job, share/install flows)
- hermes_wisdom/agent_led/: policy resolution (server > local > defaults),
7-day evidence builder that excludes bundled/hub/managed skills and
dismissed/handled/recently-suggested content hashes, strict pydantic
schemas for agent output with repair-or-reject, fixed copy templates
(Share / Teammate / Published / Update / Mute), idempotent retried
delivery ledger with stale-action resolution, weekly review job,
resumable Share and Install flows.
- prompts/: candidate review, recipient recommendation, share packaging.
- tests/wisdom/test_agent_led.py: 30 tests.
* wisdom: agent-led renderers and button action dispatcher
- render.py: Telegram HTML, Slack blocks, Desktop payload; editorial name
is the emphasized line, product label stays separate.
- actions.py: resolve opaque wa:<action>:<dedup> targets via the delivery
ledger; Not now -> dismissal, Mute -> fixed options, Share -> resumable
packaging flow, Install/Update -> plan command. Never publishes/installs.
* wisdom: CLI verbs, agent_led config default, conversational catalog skill
- hermes wisdom browse/review-week/act/share/dismiss/mute (all --json).
- wisdom.agent_led config block, default enabled.
- SKILL.md rewritten so natural-language catalog questions map to the CLI
verbs, share/install flows and fixed notification templates.
* wisdom: wire agent-led weekly review into gateway tick and Telegram buttons
- gateway housekeeping tick calls maybe_run_weekly_review with a home
channel sender when a Telegram adapter is available.
- Telegram: wa: callbacks resolved through the ledger (stale-safe), mute
duration keyboard, send_wisdom_agent_recommendation rich card + fallback.
* fix(wisdom): integrate local mediation and harden model and setup boundaries
* fix(wisdom): honor authoritative recommendation policy and defer on failure
* fix(wisdom): synchronize opaque suppression and recheck delivery preferences
* feat(wisdom): route weekly selection through the session-owned assessment queue
* fix(wisdom): prepare and submit the reviewed generated share package
* feat(wisdom): separate native Share preparation from publication consent
* feat(wisdom): sync native mute choices through a leased preference outbox
* feat(wisdom): bind native mute controls to durable preference choices
* feat(wisdom): add scoped desktop and dashboard notification settings
* fix(wisdom): revalidate feed recommendations before assessment and delivery
* fix(wisdom): persist validated delivery receipts before completing notices
* feat(wisdom): add private notification claim and receipt client
* Persist Wisdom send reservations and recover delivery acknowledgements
* Route legacy Wisdom controls through current native review
* Add typed private Wisdom operation outcome client
* fix(wisdom): make agent-led advice usable in the local demo
* fix(wisdom): keep requested consent outside proactive limits
* fix(wisdom): distinguish unavailable assessments and preserve digest text
* fix(wisdom): assess ongoing usefulness beyond the current task
* fix(wisdom): restore immediate qualification sharing controls
* fix(wisdom): separate qualification review from installation advice
* fix(wisdom): collapse review checklists and simplify sharing copy
* fix(wisdom): show compact sharing progress and publication receipts
* fix(wisdom): require credential prefixes rather than matching skill names
* fix(wisdom): finish package checks before presenting sharing consent
* fix(wisdom): scan local skills before qualification cards
* fix(wisdom): update moderation results on existing sharing cards
* fix(wisdom): keep sharing review accessible from receipt cards
* fix(wisdom): align mediated review cards and collapsible checks
* fix(wisdom): clarify clean security summary wording
* fix(wisdom): normalize consent plans and add explicit recheck
* fix(wisdom): keep install and update receipts concise
* fix(wisdom): collapse assessments and deduplicate operation cards
* fix(wisdom): restore private Portal review from native cards
* fix(wisdom): sync Portal publication to original consent card
* fix(wisdom): show local skill version on sharing cards
* fix(wisdom): skip agent recommendations for self-published versions
* fix(wisdom): simplify candidate notices and local-edit recovery copy
* feat(wisdom): submit locally reviewed packages with one confirmation
* feat(wisdom): expose safe receipt and outcome sync recovery
* wisdom: onboarding notice says detect and share, names the user's own skill
Copy review from the product owner on the first and returning
qualification notices (fixed delivery mode):
- the feature blurb now says the org enabled detection *and sharing*
- both notices say the detected skill is one the user created
- both close with an exclamation mark
Applied identically to hermes_wisdom.notice, the desktop and web i18n
strings, and the tests that assert the sentences.
* wisdom: one opener, no approval line, ask to share after the skill is shown
Product owner review of the candidate card.
- The Hermes written card now opens with the same sentence as the fixed card
("Your organisation has enabled Collective Wisdom, a feature designed to
automatically detect and share useful skills across all team members.")
instead of its own blurb, so there is one first time message.
- "Nothing is shared without your approval." removed from Telegram, Slack
and Desktop. The buttons already make the permission explicit.
- "Would you like to share?" no longer appears before the skill is named.
It is now the last line, after the skill name, description, why suggested
and the checks, and reads "Would you like to share it?" (matching the
agent led template wording).
Tests updated for the new order; proposalNotice removed from all desktop locales.
* wisdom: American spelling, organization
Product owner decision: user facing copy uses American spelling.
Changes "Your organisation" to "Your organization" in the chat notice,
the Hermes written card opener, the desktop and web strings, and the
tests that assert them. Identifiers such as nas_organisation:* and the
German and French locales are untouched.
* wisdom: candidate card copy round 4 (owner review)
Apply the product owner's round 4 copy decisions to the Hermes Collective
Wisdom candidate card on Telegram, Slack, Desktop and the shared views:
1. Hermes-written cards are titled "Hermes Collective Wisdom" instead of
the bare "Collective Wisdom".
2. The "Reusable skill ready to review" line is gone from the candidate
card (Telegram rich card and plain fallback, legacy agent-led share
template).
3. The skill name and description are labelled: "Skill name: <name>" and
"What it does: <description>" (Telegram, Slack, Desktop).
4. "Why suggested:" is now "Why others might benefit:".
5. A passing professionalism review reads "Safe to share at work ✓ (no
inappropriate content found)" with no per-check bullets and no "Pass";
a failed review reads "Needs a look before sharing at work (possible
inappropriate content)" and lists only the checks that flagged
something. Pending/unavailable wording is unchanged.
6. Telegram button toasts: "Will ask later...", "Preparing more
details...", "Sharing...".
7. Qualification reasons: "You used this skill consistently across many
days." and "You've really refined this skill."
8. prompts/wisdom_candidate_review.md asks for a compelling
editorial_name, a simple one_line_description and a compelling
why_coworkers_benefit under 300 characters; "Be concise and
convincing." becomes "Be concise and compelling: the goal is that the
user wants to share it."
Tests updated for the new strings; review_text() gains direct coverage.
* wisdom: re-apply owner copy after rebase
- Native share cards (advice_view/interaction_view): drop the approval line, ask "Would you like to share it?" as the last line after the checks
- Hermes-written completion card titled "Hermes Collective Wisdom"
- Qualification reasons use the owner wording (consistently across many days / really refined)
- American spelling (organization) in remaining English copy
- Desktop test asserts the current Share button; web test matches the returning notice
* fix(wisdom): pin reconciled Gateway and verify Unicode hash vectors
Pin Gateway 60cd2d6b613ae3cd4a6e65155d1142006d907e78 and byte-identical producer artifacts. Verify every content-order case and package-manifest binding. Validation: 186 focused Python tests, Ruff and contract verifier.
* fix(wisdom): reconcile optional SDK tests and frontend lint
* fix(wisdom): default to agent-written notification summaries
* fix(wisdom): restore deferred install review and browse controls
* feat(wisdom): inspect installed setup with exact package provenance
* feat(wisdom): run native-approved installed setup steps with durable evidence
* fix(wisdom): recover interrupted setup with explicit native consent
* feat(wisdom): hand native installs into guided setup review
* fix(wisdom): continue requested setup with fixed notification copy
* fix(wisdom): preserve setup while waiting for a session model
* fix(wisdom): expose canonical setup review controls on desktop
* fix(wisdom): resume setup after recorded automatic updates
* fix(wisdom): make missing setup prerequisites recheckable
* chore(wisdom): align Agent with verified Gateway contract
* fix(wisdom): stop guessing team slugs in portal links
* fix(wisdom): retire pending advice on account sign-out
* fix(wisdom): cancel advice after terminal account revocation
* fix(wisdom): fence feed responses across account sign-out
* fix(wisdom): checkpoint signed-out feed before reactivation
* fix(wisdom): link proactive advice to scoped notification settings
* fix(wisdom): coalesce queued publication recommendations by version
* fix(wisdom): keep package review navigation local and deferable
* fix(wisdom): reflect installed state in discovery controls
* fix(wisdom): show exact checks before command confirmation
* chore(wisdom): pin bounded analytics privacy contract
* chore(wisdom): pin retired legacy notification contract
* feat(wisdom): review publisher usage with exact sharing copy
* fix(wisdom): align discovery and review check summaries
* fix(wisdom): show expired consent before confirmation
* fix(wisdom): require fresh review for legacy install controls
* fix(wisdom): preserve review expiry across check toggles
* fix(wisdom): retain update policy in native install reviews
* fix(wisdom): surface failed native card edits
* fix(wisdom): persist local command approval reviews
* fix(wisdom): use saved approvals for messaging commands
* test(wisdom): provide scan result in setup handoff fixture
* test(wisdom): exercise Telegram approvals with saved review state
* fix(wisdom): retain suppression policy for offline deferral
* fix(wisdom): reconsider candidates after deferred suppression expires
* fix(wisdom): bind review checks and report verified readiness separately
* fix(wisdom): persist accepted publication intent and recover exact outcomes
* fix(sync): pin UTF-8 tree ordering across writers
* chore(wisdom): pin organisation-scoped Gateway authorization
* fix(wisdom): restrict consent delivery to user-facing sessions
* chore(wisdom): refresh reviewed Gateway contract pin
* fix(wisdom): preserve kept tools in Blank Slate exclusions
* test(auth): reset anonymous fixture with a profile-scoped cache
* fix(wisdom): gate local surfaces and work on current profile entitlement
* fix(wisdom): invalidate quiet tool cache on entitlement changes
* test(wisdom): authorize local consent gateway fixtures
* fix(wisdom): keep entitlement decoding free of native crypto imports
* test(wisdom): provide local entitlement to demo CLI subprocess
* ci: leave upstream workflow unchanged in Wisdom PR
* fix(wisdom): ship package and contracts in Nix wheels
---------
Co-authored-by: hbizi <36184542+hbizi@users.noreply.github.com>
605 lines
28 KiB
Python
605 lines
28 KiB
Python
"""Tests for the agent-led Collective Wisdom sharing layer."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from datetime import datetime, timedelta, timezone
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from hermes_wisdom.agent_led import templates
|
|
from hermes_wisdom.agent_led.evidence import build_evidence, provenance_exclusion
|
|
from hermes_wisdom.agent_led.history import SuggestionHistory
|
|
from hermes_wisdom.agent_led.install_flow import InstallFlow, detect_prerequisites
|
|
from hermes_wisdom.agent_led.notify import (
|
|
STALE_ACTION_MESSAGE,
|
|
DeliveryLedger,
|
|
deliver,
|
|
share_candidate_event,
|
|
teammate_event,
|
|
)
|
|
from hermes_wisdom.agent_led.policy import AgentLedPolicy, load_policy
|
|
from hermes_wisdom.agent_led.schemas import (
|
|
SchemaRejected,
|
|
parse_candidate_review,
|
|
parse_recipient_recommendation,
|
|
parse_share_package,
|
|
)
|
|
from hermes_wisdom.agent_led.share_flow import ShareFlow, scan_credentials
|
|
from hermes_wisdom.qualification import record_successful_use
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _authorize_qualification_work(monkeypatch):
|
|
from hermes_cli.config import save_config
|
|
from tests.wisdom.local_auth import authorize_local
|
|
|
|
authorize_local(monkeypatch, "org-1")
|
|
save_config({"wisdom": {"enabled": True, "disclosure_acknowledged_at": "fixture"}})
|
|
from hermes_wisdom.store import WisdomStore
|
|
|
|
NOW = datetime(2026, 3, 10, 12, 0, tzinfo=timezone.utc)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# fixtures
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def _store(tmp_path: Path, *, signed_in: bool = True) -> WisdomStore:
|
|
store = WisdomStore(tmp_path / "state")
|
|
store.installation_identity()
|
|
if signed_in:
|
|
store.verify_installation_identity("org-1")
|
|
return store
|
|
|
|
|
|
def _skill(root: Path, name: str, *, env: list[str] | None = None) -> Path:
|
|
path = root / name
|
|
path.mkdir(parents=True)
|
|
env_block = ""
|
|
if env:
|
|
env_block = "required_environment_variables: [" + ", ".join(env) + "]\n"
|
|
(path / "SKILL.md").write_text(
|
|
f"---\nname: {name}\ndescription: Use when doing {name} work.\n{env_block}---\n# Body\n",
|
|
encoding="utf-8",
|
|
)
|
|
return path
|
|
|
|
|
|
def _make_eligible(monkeypatch, skills_root: Path) -> None:
|
|
monkeypatch.setattr("hermes_wisdom.qualification.get_skills_dir", lambda: skills_root)
|
|
monkeypatch.setattr(
|
|
"hermes_wisdom.qualification._find_skill_dir", lambda name: skills_root / name
|
|
)
|
|
monkeypatch.setattr("hermes_wisdom.qualification.is_bundled", lambda _n: False)
|
|
monkeypatch.setattr("hermes_wisdom.qualification.is_hub_installed", lambda _n: False)
|
|
monkeypatch.setattr("hermes_wisdom.qualification._profile_timezone", lambda: (timezone.utc, "UTC"))
|
|
monkeypatch.setattr("hermes_wisdom.qualification.ensure_skill_editorial_metadata", lambda p: {"editorial_name": p.name, "editorial_description": "", "changed": False})
|
|
monkeypatch.setattr("hermes_wisdom.agent_led.evidence.get_skills_dir", lambda: skills_root)
|
|
monkeypatch.setattr("hermes_wisdom.agent_led.evidence.is_bundled", lambda _n: False)
|
|
monkeypatch.setattr("hermes_wisdom.agent_led.evidence.is_hub_installed", lambda _n: False)
|
|
|
|
|
|
def _use(store: WisdomStore, name: str, days_ago: list[int], per_day: int = 1) -> None:
|
|
for offset in days_ago:
|
|
for _ in range(per_day):
|
|
record_successful_use(name, store=store, at=NOW - timedelta(days=offset))
|
|
|
|
|
|
def _rec(name: str, content_hash: str, count: int = 5) -> dict:
|
|
return {
|
|
"skill_name": name,
|
|
"content_hash": content_hash,
|
|
"editorial_name": "Release Notes Drafter",
|
|
"one_line_description": "Drafts release notes from merged PRs.",
|
|
"what_it_does": "Collects merged PRs and drafts notes.",
|
|
"how_user_relied_on_it": "weekly release announcements",
|
|
"evidence": {"window_days": 7, "invocation_count": count, "days_used": 3, "last_used_at": None, "examples": []},
|
|
"why_coworkers_benefit": "every team ships weekly",
|
|
"audience": "the platform team",
|
|
"portability": [],
|
|
"confidence": 0.8,
|
|
"allowed_actions": ["share", "review", "not_now"],
|
|
}
|
|
|
|
|
|
def _review_json(name: str, content_hash: str, count: int = 5) -> str:
|
|
return json.dumps(
|
|
{
|
|
"schema_version": 1,
|
|
"window_days": 7,
|
|
"considered": [name],
|
|
"recommendations": [_rec(name, content_hash, count)],
|
|
"nothing_to_recommend_reason": None,
|
|
"requires_confirmation": True,
|
|
}
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# policy
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_policy_defaults_and_local_override(monkeypatch):
|
|
monkeypatch.setattr("hermes_wisdom.mediation.delivery_mode", lambda: "agent")
|
|
policy = load_policy(local={})
|
|
assert policy.enabled is False # no authoritative organization policy
|
|
assert policy.window_days == 7
|
|
assert policy.min_aggregate_count == 3
|
|
assert policy.dismiss_suppression_days == 30
|
|
assert policy.popular_install_threshold == 10
|
|
custom = load_policy(local={"enabled": False, "window_days": 14, "min_aggregate_count": 0})
|
|
assert custom.enabled is False
|
|
assert custom.window_days == 14
|
|
assert custom.min_aggregate_count == 1 # floor
|
|
assert custom.source == "local_config"
|
|
|
|
|
|
def test_policy_server_block_wins(monkeypatch):
|
|
from hermes_wisdom.client import AgentLedPolicyResponse
|
|
|
|
monkeypatch.setattr("hermes_wisdom.mediation.delivery_mode", lambda: "agent")
|
|
class Client:
|
|
display_org_id = "org-1"
|
|
|
|
def agent_led_policy(self):
|
|
return AgentLedPolicyResponse(
|
|
org_id="org-1", usage_evidence_window_days=10,
|
|
min_aggregate_invocations=8, consecutive_day_usage_counts=False,
|
|
repeated_edits_count=False, max_recommendations_per_user_per_week=0,
|
|
publication_mode="moderated", install_popularity_threshold=20,
|
|
notification_defaults={"skill_ready_to_share": False, "teammate_published": True, "update_available": False},
|
|
manager_review_email_cadence="daily", not_now_suppression_days=60,
|
|
version=1, updated_by_user_id=None,
|
|
)
|
|
|
|
policy = load_policy(client=Client(), local={"window_days": 5})
|
|
assert policy.window_days == 10
|
|
assert policy.source == "server_policy"
|
|
assert policy.enabled is True
|
|
assert policy.min_aggregate_count == 8
|
|
assert policy.max_candidates == 0
|
|
assert policy.dismiss_suppression_days == 60
|
|
assert policy.notification_defaults["skill_ready_to_share"] is False
|
|
assert policy.consecutive_day_usage_counts is False
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# evidence: built-in exclusion + 7-day aggregate counting
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_builtin_skills_excluded(monkeypatch, tmp_path):
|
|
skills = tmp_path / "skills"
|
|
_skill(skills, "own-skill")
|
|
_skill(skills, "bundled-skill")
|
|
_make_eligible(monkeypatch, skills)
|
|
store = _store(tmp_path)
|
|
_use(store, "own-skill", [0, 1, 2], per_day=2)
|
|
_use(store, "bundled-skill", [0, 1, 2], per_day=2)
|
|
monkeypatch.setattr(
|
|
"hermes_wisdom.agent_led.evidence.is_bundled", lambda n: n == "bundled-skill"
|
|
)
|
|
payload = build_evidence(
|
|
store=store, policy=AgentLedPolicy(), history=SuggestionHistory(tmp_path / "h.json"), at=NOW, skills_root=skills
|
|
)
|
|
names = [c.skill_name for c in payload.candidates]
|
|
assert names == ["own-skill"]
|
|
assert payload.excluded["bundled"] == ["bundled-skill"]
|
|
assert provenance_exclusion("x", skills / "_wisdom" / "org" / "x", skills_root=skills) == "managed_or_archived"
|
|
|
|
|
|
def test_seven_day_aggregate_counting(monkeypatch, tmp_path):
|
|
skills = tmp_path / "skills"
|
|
_skill(skills, "counted", env=["MY_TOKEN"])
|
|
_make_eligible(monkeypatch, skills)
|
|
store = _store(tmp_path)
|
|
# 2 uses/day on 3 distinct days inside the window, plus 4 uses outside it.
|
|
_use(store, "counted", [0, 3, 6], per_day=2)
|
|
_use(store, "counted", [7, 8], per_day=2)
|
|
payload = build_evidence(
|
|
store=store, policy=AgentLedPolicy(), history=SuggestionHistory(tmp_path / "h.json"), at=NOW, skills_root=skills
|
|
)
|
|
assert len(payload.candidates) == 1
|
|
candidate = payload.candidates[0]
|
|
assert candidate.invocation_count == 6
|
|
assert candidate.days_used == 3
|
|
assert candidate.last_used_day == NOW.date().isoformat()
|
|
assert candidate.required_environment_variables == ["MY_TOKEN"]
|
|
assert payload.window_start == (NOW.date() - timedelta(days=6)).isoformat()
|
|
|
|
|
|
def test_below_min_count_excluded(monkeypatch, tmp_path):
|
|
skills = tmp_path / "skills"
|
|
_skill(skills, "rare")
|
|
_make_eligible(monkeypatch, skills)
|
|
store = _store(tmp_path)
|
|
_use(store, "rare", [0])
|
|
payload = build_evidence(
|
|
store=store, policy=AgentLedPolicy(min_aggregate_count=3), history=SuggestionHistory(tmp_path / "h.json"), at=NOW, skills_root=skills
|
|
)
|
|
assert payload.candidates == []
|
|
assert payload.excluded["below_min_count"] == ["rare"]
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# dismissal suppression by hash
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_dismissal_suppresses_same_hash_only(tmp_path):
|
|
history = SuggestionHistory(tmp_path / "h.json")
|
|
result = history.record_dismissal("s", "hash-a", suppression_days=30, at=NOW)
|
|
assert result["gateway"] == "pending_gateway"
|
|
assert history.is_suppressed("s", "hash-a", at=NOW + timedelta(days=29))
|
|
assert not history.is_suppressed("s", "hash-a", at=NOW + timedelta(days=31))
|
|
assert not history.is_suppressed("s", "hash-b", at=NOW) # content changed
|
|
reloaded = SuggestionHistory(tmp_path / "h.json")
|
|
assert reloaded.is_suppressed("s", "hash-a", at=NOW)
|
|
assert reloaded.pending_gateway()[0]["kind"] == "dismissal"
|
|
|
|
|
|
def test_dismissed_skill_excluded_from_evidence(monkeypatch, tmp_path):
|
|
skills = tmp_path / "skills"
|
|
_skill(skills, "dismissed")
|
|
_make_eligible(monkeypatch, skills)
|
|
store = _store(tmp_path)
|
|
_use(store, "dismissed", [0, 1, 2], per_day=2)
|
|
history = SuggestionHistory(tmp_path / "h.json")
|
|
first = build_evidence(store=store, policy=AgentLedPolicy(), history=history, at=NOW, skills_root=skills)
|
|
history.record_dismissal("dismissed", first.candidates[0].content_hash, at=NOW)
|
|
second = build_evidence(store=store, policy=AgentLedPolicy(), history=history, at=NOW, skills_root=skills)
|
|
assert second.candidates == []
|
|
assert second.excluded["dismissed"] == ["dismissed"]
|
|
|
|
|
|
def test_mute_uses_fixed_options(tmp_path):
|
|
history = SuggestionHistory(tmp_path / "h.json")
|
|
assert [label for _k, label, _d in templates.MUTE_OPTIONS] == ["1 day", "1 week", "30 days", "Forever"]
|
|
history.record_mute("*", days=templates.mute_duration_days("1w"), at=NOW)
|
|
assert history.is_muted(at=NOW + timedelta(days=6))
|
|
assert not history.is_muted(at=NOW + timedelta(days=8))
|
|
history.record_mute("skill-x", days=templates.mute_duration_days("forever"), at=NOW)
|
|
assert history.is_muted("skill-x", at=NOW + timedelta(days=4000))
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# schema validation
|
|
|
|
|
|
def test_credential_detector_allows_long_skill_names():
|
|
from hermes_wisdom.agent_led.schemas import PackagedFile
|
|
text = "Use skill-qualification-test and skill-publishing-flow-test."
|
|
assert PackagedFile(path="SKILL.md", content=text).content == text
|
|
|
|
|
|
@pytest.mark.parametrize("prefix", ["sk-", "ghp_", "gho_", "xoxb-", "AKIA", "AIza"])
|
|
def test_credential_detector_still_rejects_key_shapes(prefix):
|
|
from hermes_wisdom.agent_led.schemas import PackagedFile
|
|
with pytest.raises(ValueError, match="credential-shaped"):
|
|
PackagedFile(path="SKILL.md", content=prefix + "a" * 32)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_candidate_review_schema_accepts_valid_and_repairs_bad_item():
|
|
good = parse_candidate_review(_review_json("s", "abcdef1234"))
|
|
assert good.recommendations[0].skill_name == "s"
|
|
assert good.requires_confirmation is True
|
|
payload = json.loads(_review_json("s", "abcdef1234"))
|
|
payload["recommendations"].append({"skill_name": "broken"}) # invalid item
|
|
repaired = parse_candidate_review(json.dumps(payload))
|
|
assert [r.skill_name for r in repaired.recommendations] == ["s"]
|
|
|
|
|
|
def test_candidate_review_schema_rejects_secrets_and_garbage():
|
|
payload = json.loads(_review_json("s", "abcdef1234"))
|
|
payload["recommendations"][0]["what_it_does"] = "uses token ghp_" + "a" * 36
|
|
# A credential-shaped recommendation is dropped by the repair pass, never surfaced.
|
|
assert parse_candidate_review(json.dumps(payload)).recommendations == []
|
|
with pytest.raises(SchemaRejected):
|
|
parse_candidate_review(json.dumps({**payload, "window_days": 0}))
|
|
with pytest.raises(SchemaRejected):
|
|
parse_candidate_review("not json at all")
|
|
with pytest.raises(SchemaRejected):
|
|
parse_candidate_review(json.dumps({**json.loads(_review_json("s", "abcdef1234")), "requires_confirmation": False}))
|
|
|
|
|
|
def test_candidate_review_never_removes_not_now():
|
|
payload = json.loads(_review_json("s", "abcdef1234"))
|
|
payload["recommendations"][0]["allowed_actions"] = ["share"]
|
|
parsed = parse_candidate_review(json.dumps(payload))
|
|
assert "not_now" in parsed.recommendations[0].allowed_actions
|
|
|
|
|
|
def test_recipient_recommendation_schema():
|
|
irrelevant = parse_recipient_recommendation(
|
|
json.dumps({"schema_version": 1, "skill_id": "sk", "version": 1, "relevant": False, "not_relevant_reason": "no overlap", "confidence": 0.9})
|
|
)
|
|
assert irrelevant.relevant is False and "mute" in irrelevant.allowed_actions
|
|
with pytest.raises(SchemaRejected):
|
|
parse_recipient_recommendation(
|
|
json.dumps({"schema_version": 1, "skill_id": "sk", "version": 1, "relevant": True, "confidence": 0.9})
|
|
)
|
|
with pytest.raises(SchemaRejected):
|
|
parse_recipient_recommendation(
|
|
json.dumps({"schema_version": 1, "skill_id": "sk", "version": 1, "relevant": False, "confidence": 0.9, "installed": True})
|
|
)
|
|
|
|
|
|
def test_share_package_schema_requires_skill_md_and_safe_paths():
|
|
base = {
|
|
"schema_version": 1, "skill_name": "s", "source_content_hash": "abcdef1234", "editorial_name": "S",
|
|
"plain_description": "Does S.", "files": [{"path": "SKILL.md", "content": "# S"}],
|
|
"requirements": [], "setup_instructions": [], "credential_handoff": [], "compatibility_limits": [],
|
|
"verification_step": "hermes skills list | grep S", "removed_or_generalized": [], "related_skills": [],
|
|
}
|
|
assert parse_share_package(json.dumps(base)).skill_name == "s"
|
|
with pytest.raises(SchemaRejected):
|
|
parse_share_package(json.dumps({**base, "files": [{"path": "README.md", "content": "x"}]}))
|
|
with pytest.raises(SchemaRejected):
|
|
parse_share_package(json.dumps({**base, "files": [{"path": "../SKILL.md", "content": "x"}]}))
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# templates: exact headers, no 'Pass'
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_share_template_exact_strings():
|
|
notice = templates.render_share(
|
|
editorial_name="Release Notes Drafter",
|
|
description="Drafts release notes from merged PRs.",
|
|
count_7d=9,
|
|
specific_work="weekly release announcements",
|
|
audience="the platform team",
|
|
reason="every team ships weekly",
|
|
checks_passed=True,
|
|
)
|
|
text = notice.text
|
|
assert notice.title == "Hermes Collective Wisdom"
|
|
assert "Reusable skill ready to review" not in text
|
|
assert notice.lines[0] == "Release Notes Drafter"
|
|
assert notice.lines[1] == " Drafts release notes from merged PRs."
|
|
assert "Why we ask" in notice.lines
|
|
assert (
|
|
"You used this skill 9 times in the last 7 days. It has helped with weekly release "
|
|
"announcements. I think it could help the platform team because every team ships weekly."
|
|
) in notice.lines
|
|
assert "Checks complete: no issues detected." in notice.lines
|
|
for check in (
|
|
"No profanity or abusive language",
|
|
"No hate or harassment",
|
|
"No sexual or graphic content",
|
|
"No detected credentials or private keys",
|
|
):
|
|
assert f"✓ {check}" in notice.lines
|
|
assert notice.lines[-1] == "Would you like to share it?"
|
|
assert [a.label for a in notice.actions] == ["Not now", "Review", "Share"]
|
|
assert "Pass" not in text.split()
|
|
assert "Pass" not in text
|
|
|
|
|
|
def test_share_template_requires_real_count():
|
|
with pytest.raises(ValueError):
|
|
templates.render_share(
|
|
editorial_name="X", description="d", count_7d=None, # type: ignore[arg-type]
|
|
specific_work="w", audience="a", reason="r",
|
|
)
|
|
|
|
|
|
def test_teammate_template_exact_strings():
|
|
notice = templates.render_teammate(
|
|
editorial_name="Incident Timeline Builder",
|
|
outcome_one_liner="Turn a noisy incident channel into a clean timeline.",
|
|
publisher_name="Dana",
|
|
specific_work="post-incident reviews",
|
|
count_7d=12,
|
|
recipient_reason="your on-call rotation",
|
|
installation_count=14,
|
|
)
|
|
assert notice.title == "New skill from your team"
|
|
assert notice.lines[0] == "Incident Timeline Builder"
|
|
assert notice.lines[2] == "Published by Dana"
|
|
assert (
|
|
"Dana uses this for post-incident reviews. It was used 12 times in the last 7 days. "
|
|
"I think it could help you with your on-call rotation."
|
|
) in notice.lines
|
|
assert "✓ Security check complete (credentials, private keys, organization policy)" in notice.lines
|
|
assert "Popular with your team: 10+ installations" in notice.lines
|
|
assert [a.label for a in notice.actions] == ["Mute", "View", "Install"]
|
|
assert "New notification" not in notice.text
|
|
assert "Pass" not in notice.text
|
|
quiet = templates.render_teammate(
|
|
editorial_name="X", outcome_one_liner="o", publisher_name="P", specific_work="w",
|
|
count_7d=1, recipient_reason="r", installation_count=3,
|
|
)
|
|
assert not any(line.startswith("Popular with your team") for line in quiet.lines)
|
|
|
|
|
|
def test_published_and_update_templates():
|
|
open_notice = templates.render_published(organization_name="Acme", moderated=False)
|
|
assert open_notice.title == "Published to Acme."
|
|
assert open_notice.lines == ["Your teammates can now find and install this skill."]
|
|
assert [a.label for a in open_notice.actions] == ["View in Portal"]
|
|
moderated = templates.render_published(organization_name="Acme", moderated=True)
|
|
assert moderated.title == "Sent for review."
|
|
assert moderated.lines == [
|
|
"It is not available to your organization yet. You will be notified when the review is complete."
|
|
]
|
|
update = templates.render_update(skill_name="Release Notes Drafter", summary="Adds changelog links.", reason="you ship weekly", setup_impact=None)
|
|
assert update.title == "Update available for Release Notes Drafter"
|
|
assert update.lines == [
|
|
"Adds changelog links.",
|
|
"This matters for your work because you ship weekly.",
|
|
"Setup impact: None",
|
|
]
|
|
assert [a.label for a in update.actions] == ["Mute", "View changes", "Update"]
|
|
for notice in (open_notice, moderated, update):
|
|
assert "Pass" not in notice.text
|
|
|
|
|
|
def test_templates_reject_forbidden_words_in_prose():
|
|
with pytest.raises(ValueError):
|
|
templates.render_update(skill_name="S", summary="Pass", reason="r")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# idempotent delivery + stale buttons
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_legacy_delivery_cannot_bypass_native_ownership(tmp_path):
|
|
from unittest.mock import Mock
|
|
from hermes_wisdom.agent_led.notify import DeliveryError
|
|
from hermes_wisdom.agent_led.schemas import CandidateRecommendation
|
|
|
|
rec = CandidateRecommendation.model_validate(_rec("s", "abcdef1234"))
|
|
event = share_candidate_event(rec, organization_id="org-1", recipient_id="u1", at=NOW)
|
|
sender = Mock()
|
|
ledger = DeliveryLedger(tmp_path / "ledger.json")
|
|
with pytest.raises(DeliveryError, match="profile-owned mediation"):
|
|
deliver(event, sender=sender, ledger=ledger, retries=2, sleep=lambda _s: None, now=NOW)
|
|
sender.assert_not_called()
|
|
assert ledger.get(event.dedup_key) is None
|
|
# Same inputs -> same dedup key across processes.
|
|
again = share_candidate_event(rec, organization_id="org-1", recipient_id="u1", at=NOW)
|
|
assert again.dedup_key == event.dedup_key
|
|
|
|
|
|
def test_legacy_delivery_does_not_modify_existing_snapshots(tmp_path):
|
|
from unittest.mock import Mock
|
|
from hermes_wisdom.agent_led.notify import DeliveryError
|
|
from hermes_wisdom.agent_led.schemas import CandidateRecommendation
|
|
|
|
rec = CandidateRecommendation.model_validate(_rec("s", "abcdef1234"))
|
|
event = share_candidate_event(rec, organization_id="org-1", recipient_id="u1", at=NOW)
|
|
ledger = DeliveryLedger(tmp_path / "ledger.json")
|
|
|
|
ledger.record(event, state="delivered")
|
|
before = ledger.path.read_bytes()
|
|
sender = Mock()
|
|
with pytest.raises(DeliveryError):
|
|
deliver(event, sender=sender, ledger=ledger, now=NOW)
|
|
sender.assert_not_called()
|
|
assert ledger.path.read_bytes() == before
|
|
|
|
|
|
def test_stale_and_duplicate_actions(tmp_path):
|
|
from hermes_wisdom.agent_led.schemas import CandidateRecommendation
|
|
|
|
rec = CandidateRecommendation.model_validate(_rec("s", "abcdef1234"))
|
|
event = share_candidate_event(rec, organization_id="org-1", recipient_id="u1", at=NOW, ttl_hours=1)
|
|
ledger = DeliveryLedger(tmp_path / "ledger.json")
|
|
ledger.record(event, state="delivered")
|
|
share_target = next(a.target for a in event.allowed_actions if a.id == "share")
|
|
resolved = ledger.resolve_action(share_target, at=NOW)
|
|
assert resolved["ok"] and resolved["action"] == "share"
|
|
assert ledger.mark_acted(event.dedup_key, "share") is True
|
|
assert ledger.mark_acted(event.dedup_key, "share") is False # no duplicate action
|
|
assert ledger.resolve_action(share_target, at=NOW)["message"] == STALE_ACTION_MESSAGE
|
|
fresh = share_candidate_event(rec, organization_id="org-2", recipient_id="u1", at=NOW, ttl_hours=1)
|
|
ledger.record(fresh, state="delivered")
|
|
expired = ledger.resolve_action(fresh.allowed_actions[0].target, at=NOW + timedelta(hours=2))
|
|
assert expired["stale"] and expired["message"] == STALE_ACTION_MESSAGE
|
|
assert ledger.resolve_action("garbage", at=NOW)["stale"]
|
|
|
|
|
|
def test_teammate_event_requires_relevance():
|
|
from hermes_wisdom.agent_led.schemas import RecipientRecommendation
|
|
|
|
rec = RecipientRecommendation.model_validate(
|
|
{"schema_version": 1, "skill_id": "sk", "version": 1, "relevant": False, "confidence": 0.5}
|
|
)
|
|
with pytest.raises(ValueError):
|
|
teammate_event(rec, organization_id="o", recipient_id="r")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# share flow: no publish without confirmation
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def _package_json(name: str, content_hash: str, *, leak: bool = False) -> str:
|
|
body = "# Notes\n" + ("token = ghp_" + "b" * 36 if leak else "Generalized body.")
|
|
return json.dumps(
|
|
{
|
|
"schema_version": 1, "skill_name": name, "source_content_hash": content_hash,
|
|
"editorial_name": "Release Notes", "plain_description": "Drafts release notes.",
|
|
"files": [{"path": "SKILL.md", "content": body, "generalized_from_original": True}],
|
|
"requirements": [{"kind": "command", "name": "gh", "purpose": "read merged PRs", "handoff": "install GitHub CLI"}],
|
|
"setup_instructions": ["Install gh", "Authenticate gh"],
|
|
"credential_handoff": ["A GitHub token with repo read scope, from your own account"],
|
|
"compatibility_limits": ["Requires GitHub-hosted repositories"],
|
|
"verification_step": "gh --version",
|
|
"removed_or_generalized": ["Replaced a home directory path with $HOME"],
|
|
"related_skills": [],
|
|
}
|
|
)
|
|
|
|
|
|
def test_share_flow_never_publishes_without_approval(tmp_path):
|
|
skill = _skill(tmp_path / "skills", "notes")
|
|
(skill / "SKILL.md").write_text(
|
|
(skill / "SKILL.md").read_text() + "\nSee /home/alice/notes and token = ghp_" + "c" * 36 + "\n"
|
|
)
|
|
flow = ShareFlow(root=tmp_path / "flows")
|
|
summary = flow.start(skill)
|
|
assert summary["status"] == "prepass"
|
|
assert any("Credential-shaped" in p for p in summary["portability_problems"])
|
|
assert any("Organization-specific" in p for p in summary["portability_problems"])
|
|
published = {"n": 0}
|
|
|
|
def submit(package, _state):
|
|
published["n"] += 1
|
|
return {"draft_id": "d1", "published": False}
|
|
|
|
with pytest.raises(RuntimeError):
|
|
flow.approve(submit=submit) # not packaged yet
|
|
assert published["n"] == 0
|
|
content_hash = summary["source_content_hash"]
|
|
flow.package(model_call=lambda m, s: _package_json("notes", content_hash))
|
|
assert flow.status == "awaiting_approval"
|
|
assert flow.summary()["next_actions"] == ["approve", "request_changes", "cancel"]
|
|
assert published["n"] == 0 # packaging alone publishes nothing
|
|
resumed = ShareFlow(flow.flow_id, root=tmp_path / "flows")
|
|
assert resumed.status == "awaiting_approval"
|
|
resumed.request_changes("mention rate limits")
|
|
assert resumed.status == "changes_requested"
|
|
resumed.package(model_call=lambda m, s: _package_json("notes", content_hash))
|
|
resumed.approve(submit=submit)
|
|
assert published["n"] == 1 and resumed.status == "submitted"
|
|
assert resumed.summary()["published"] is False # draft only; owner review still required
|
|
|
|
|
|
def test_share_flow_rejects_leaked_credentials_in_package(tmp_path):
|
|
skill = _skill(tmp_path / "skills", "notes")
|
|
flow = ShareFlow(root=tmp_path / "flows")
|
|
content_hash = flow.start(skill)["source_content_hash"]
|
|
with pytest.raises(SchemaRejected):
|
|
flow.package(model_call=lambda m, s: _package_json("notes", content_hash, leak=True))
|
|
assert flow.status == "prepass"
|
|
assert scan_credentials([{"path": "a", "content": "AKIA" + "A" * 16}])[0]["kind"] == "aws_access_key"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# install flow: success only after verification, resumable
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_install_flow_reports_installed_only_after_verification(tmp_path):
|
|
package = json.loads(_package_json("notes", "abcdef1234"))
|
|
package["requirements"].append({"kind": "env_var", "name": "NOTES_TOKEN", "purpose": "auth"})
|
|
flow = InstallFlow(root=tmp_path / "installs")
|
|
flow.start(skill_id="sk-1", version=2, package=package)
|
|
summary = flow.check_prerequisites(env={})
|
|
assert summary["step"] == "setup" and any(m["name"] == "NOTES_TOKEN" for m in summary["missing"])
|
|
assert summary["installed"] is False
|
|
resumed = InstallFlow(flow.flow_id, root=tmp_path / "installs")
|
|
assert resumed.step == "setup"
|
|
resumed.mark_setup_complete("NOTES_TOKEN")
|
|
for item in resumed.state["prerequisites"]:
|
|
item["status"] = "present"
|
|
resumed.state["step"] = "apply"
|
|
resumed.apply(lambda _state: {"installed": True, "managed_path": "x"})
|
|
assert resumed.step == "verify" and resumed.installed is False
|
|
failed = resumed.verify(runner=lambda cmd: (False, "boom"))
|
|
assert failed["installed"] is False and failed["files_installed"] is True
|
|
assert failed["message"] == "Files installed; setup verification is incomplete."
|
|
passed = resumed.verify(runner=lambda cmd: (True, "ok"))
|
|
assert passed["installed"] is True and passed["message"] == "Installed and verified."
|
|
assert detect_prerequisites([{"kind": "command", "name": "definitely-not-a-real-binary-xyz"}])[0]["status"] == "missing"
|