feat(skills): add grounded-citations skill for verifiable sourcing
Answers and written deliverables that rest on retrieved information now get inline numbered citations plus a mechanically-rendered Sources list, with a persistent ledger that makes a hallucinated citation detectable. - skills/research/grounded-citations/scripts/sources.py: stdlib citation ledger (add/ingest/list/render/verify) at $HERMES_HOME/cache/citations/ledger.json, profile-aware, O_EXCL-locked so parallel subagents sharing a ledger can't collide on ids - SKILL.md: cite-while-drafting procedure, register-at-retrieval rule, pitfalls, verification gate - references/citation-formats.md: per-target placement (markdown, LaTeX/PDF, docx, pptx, xlsx, wiki, BibTeX handoff to research-paper-writing) - references/grounding-rationale.md: why numbered ids (ALCE 2305.14627, WebGPT 2112.09332, Perplexity marker conventions), and how this relates to the in-process registry in PR #44833 - tests/skills/test_grounded_citations_skill.py: 30 tests
This commit is contained in:
@@ -0,0 +1,153 @@
|
||||
---
|
||||
name: grounded-citations
|
||||
description: "Ground answers and documents in cited, verifiable sources."
|
||||
version: 1.0.0
|
||||
author: Hermes Agent + Teknium
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [Research, Citations, Grounding, Sources, Web, Reports]
|
||||
category: research
|
||||
related_skills: [research-paper-writing, arxiv, ocr-and-documents]
|
||||
---
|
||||
|
||||
# Grounded Citations
|
||||
|
||||
Every claim taken from an outside source gets an inline numbered citation and a
|
||||
`Sources:` list, Perplexity-style. A ledger script owns the `url → [n]` mapping
|
||||
so the numbers and URLs come from retrieval, never from memory — the model only
|
||||
ever emits small integers it was handed.
|
||||
|
||||
This skill covers answers in chat, written documents (markdown, PDF, docx,
|
||||
slides), and research reports. It does not cover academic BibTeX pipelines —
|
||||
for conference papers use the `research-paper-writing` skill, which this skill
|
||||
feeds (see `references/citation-formats.md`).
|
||||
|
||||
## When to Use
|
||||
|
||||
Use whenever an answer or artifact rests on information you fetched rather than
|
||||
knew:
|
||||
|
||||
- Research, comparisons, news summaries, "what is the current state of X"
|
||||
- Any deliverable you write to disk that quotes, paraphrases, or reports
|
||||
outside facts — reports, briefs, docs, decks, wiki pages
|
||||
- Fact-finding where the user will want to check your work
|
||||
- Multi-source synthesis where conflicting sources must be attributed
|
||||
|
||||
Skip inline citations when the retrieval is incidental to another task — a
|
||||
quick syntax/version lookup mid-coding, casual conversation, creative writing.
|
||||
Mention a URL only if the user would plausibly want the link.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
None beyond the standard toolset. `scripts/sources.py` is stdlib-only Python 3.
|
||||
Retrieval comes from whatever is configured: `web_search`, `web_extract`,
|
||||
`browser_navigate`, or `terminal` (curl, CLIs).
|
||||
|
||||
Ledger location: `$HERMES_HOME/cache/citations/ledger.json` (profile-aware).
|
||||
Override per task with `--ledger <path>` or `HERMES_CITATION_LEDGER`.
|
||||
|
||||
## How to Run
|
||||
|
||||
```bash
|
||||
S=~/.hermes/skills/research/grounded-citations/scripts/sources.py
|
||||
|
||||
python3 "$S" reset # start a clean ledger
|
||||
python3 "$S" add https://example.com/a --title "A" # prints: [1]
|
||||
python3 "$S" add https://example.com/b --title "B" # prints: [2]
|
||||
python3 "$S" list # ledger table
|
||||
python3 "$S" render # Sources: block
|
||||
python3 "$S" verify draft.md # catch bad citations
|
||||
```
|
||||
|
||||
`add` is idempotent and URL-normalized: the same page always returns the same
|
||||
id within a ledger, so ids stay stable across many search/extract rounds.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Action | Command |
|
||||
|---|---|
|
||||
| Fresh ledger for a new task | `sources.py reset` |
|
||||
| Register a source, get its id | `sources.py add <url> [--title T]` |
|
||||
| Register several at once | `sources.py add <url1> <url2> ...` |
|
||||
| Register from JSON tool output | `sources.py ingest results.json` |
|
||||
| Show ledger | `sources.py list [--json]` |
|
||||
| Render the Sources block | `sources.py render [--style markdown\|plain\|footnotes\|bibtex] [--only 1,3]` |
|
||||
| Render only what a draft cites | `sources.py render --cited-in draft.md` |
|
||||
| Check a draft's citations | `sources.py verify draft.md [--strict] [--min-coverage 0.6]` |
|
||||
|
||||
## Procedure
|
||||
|
||||
① **Reset the ledger** at the start of a task that will produce a grounded
|
||||
answer or document. Skip the reset when continuing work whose ids are already
|
||||
in a draft — reusing the ledger keeps the numbering stable.
|
||||
|
||||
② **Register every source at retrieval time.** After each `web_search` /
|
||||
`web_extract` / `browser_navigate` / fetch, pass the URLs to `sources.py add`
|
||||
(or pipe the raw JSON through `sources.py ingest`). Do this *before* writing
|
||||
prose. Registering later, from memory, is the failure mode this skill exists to
|
||||
prevent.
|
||||
|
||||
③ **Write cite-while-drafting.** Place the bracketed id(s) immediately after
|
||||
each sentence the source supports:
|
||||
|
||||
```
|
||||
Ice floats because it is less dense than liquid water.[1][2]
|
||||
```
|
||||
|
||||
- No space before the bracket; each id in its own brackets.
|
||||
- Max 3 ids per sentence. Cite per sentence, not one dump at the end.
|
||||
- Only ids the ledger returned. Never invent an id or a URL.
|
||||
- Claims from your own knowledge get no citation.
|
||||
- Conflicting sources: present both readings, each with its own id.
|
||||
- Quote exact figures, dates, and names as the source states them; flag gaps
|
||||
explicitly ("no source found for X") instead of smoothing them over.
|
||||
|
||||
④ **Append the Sources block** with `sources.py render --cited-in <draft>` so
|
||||
the id → URL mapping is generated mechanically from the ledger, not retyped.
|
||||
For non-markdown targets pick the matching `--style` and follow
|
||||
`references/citation-formats.md` for placement (footnotes in docx, endnotes in
|
||||
PDF/LaTeX, a Sources slide in decks, per-page source lists in wiki output).
|
||||
|
||||
⑤ **Verify before delivering** — `sources.py verify <draft>` exits non-zero on
|
||||
unknown ids, on a Sources block that disagrees with the ledger, or (with
|
||||
`--min-coverage`) on prose that is too thinly cited. Fix and re-run.
|
||||
|
||||
⑥ **Chat answers** follow the same steps with the draft in your reply: register
|
||||
sources, cite inline, end with the rendered `Sources:` list. For a short answer
|
||||
you may render the block from `sources.py render --only <ids>` instead of
|
||||
writing to a file.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Registering after writing.** The ledger must be populated from tool output,
|
||||
not reconstructed from the draft — that reintroduces exactly the hallucinated
|
||||
-URL risk the numbering removes.
|
||||
- **Renumbering mid-task.** Never hand-edit ids in a draft. Ids are ledger
|
||||
identities; if a draft cites `[4]`, `[4]` must stay that source. Run `reset`
|
||||
only between tasks.
|
||||
- **Retyping URLs into the Sources block.** Always `render`. A hand-typed URL
|
||||
is an unverified claim.
|
||||
- **Citing a search snippet as if you read the page.** A `web_search`
|
||||
description supports only what it literally says. Cite the extracted page
|
||||
when the claim needs the body — `web_extract` it first.
|
||||
- **Over-citing.** Three ids on a sentence is the ceiling; a citation on every
|
||||
clause makes text unreadable and hides which source carries the load.
|
||||
- **Citing the ledger in code/config artifacts.** Source comments belong in
|
||||
prose deliverables and doc headers, not inside generated code.
|
||||
- **Parallel subagents.** Each subagent has its own working directory; point
|
||||
them all at one ledger with `--ledger` (or `HERMES_CITATION_LEDGER`) if their
|
||||
outputs get merged, otherwise their ids will collide.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
python3 "$S" verify report.md --strict --min-coverage 0.5
|
||||
```
|
||||
|
||||
Green means: every `[n]` in the draft exists in the ledger, the Sources block
|
||||
lists exactly the cited ids with the ledger's URLs, and the cited share of
|
||||
source-bearing sentences meets the threshold. Read the warnings even when the
|
||||
exit code is 0 — uncited registered sources usually mean a claim lost its
|
||||
attribution during editing.
|
||||
@@ -0,0 +1,65 @@
|
||||
# Citation formats per output target
|
||||
|
||||
The ledger is format-agnostic: `sources.py render --style ...` emits the block,
|
||||
this file says where it goes and what the inline marker looks like.
|
||||
|
||||
## Markdown / chat answers
|
||||
|
||||
Inline `[n]` immediately after the sentence. Block at the end:
|
||||
|
||||
```
|
||||
## Sources
|
||||
|
||||
[1] https://example.com/a — Page title
|
||||
[2] https://example.com/b
|
||||
```
|
||||
|
||||
`--style plain` gives a bare `Sources:` header for chat replies where a
|
||||
markdown heading would be noise.
|
||||
|
||||
## PDF via LaTeX (`latex-pdf-report` skill)
|
||||
|
||||
Use `--style footnotes` and map each id to `\footnote{}` at first use, or keep
|
||||
numeric markers and emit an endnotes section. For a bibliography-shaped report,
|
||||
`--style bibtex` writes `@misc` entries keyed `source<N>`; cite them with
|
||||
`\cite{source3}` and let BibTeX render the list.
|
||||
|
||||
Do not mix: either numeric `[n]` + a Sources section, or `\cite{}` + BibTeX.
|
||||
Two numbering systems in one document is worse than none.
|
||||
|
||||
## Word (.docx, `docx` skill)
|
||||
|
||||
Real footnotes are preferred over inline brackets in prose documents intended
|
||||
for human editing — reviewers expect Word footnotes. Keep the ledger ids as the
|
||||
footnote numbers so `verify` still works on a markdown source-of-truth, and
|
||||
generate the .docx from that markdown.
|
||||
|
||||
## Slides (.pptx, `powerpoint` skill)
|
||||
|
||||
Inline `[n]` in the bullet, one "Sources" slide at the end rendered with
|
||||
`--style plain`. Never put a URL in a body bullet — it wrecks the layout and
|
||||
can't be clicked in a projected deck.
|
||||
|
||||
## Spreadsheets (.xlsx)
|
||||
|
||||
Add a `source` column holding the id, plus a `Sources` sheet built from
|
||||
`render --style plain`. Do not paste URLs into data cells.
|
||||
|
||||
## Wiki / multi-page output (`llm-wiki`, Obsidian)
|
||||
|
||||
Per-page Sources block, ids shared across pages from one ledger. Because ids
|
||||
are ledger identities, `[7]` means the same page everywhere in the wiki — that
|
||||
consistency is the reason not to reset the ledger between pages of one build.
|
||||
|
||||
## Research papers
|
||||
|
||||
Hand off to the `research-paper-writing` skill. Export with
|
||||
`--style bibtex` into `references.bib`, then follow that skill's citation
|
||||
verification (it greps `\cite{...}` against the .bib). The ledger's job ends at
|
||||
producing verified URL entries; venue formatting is that skill's domain.
|
||||
|
||||
## Code and config artifacts
|
||||
|
||||
No citations inside generated code. If provenance matters, put it in the
|
||||
commit message, the PR body, or a doc header — not in comments scattered
|
||||
through source.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Why numbered ledger ids (grounding research basis)
|
||||
|
||||
Design notes for anyone changing the citation instructions or the ledger
|
||||
mechanics. The wording in SKILL.md is not arbitrary.
|
||||
|
||||
## The structural trick
|
||||
|
||||
Hallucinated citations happen when a model reconstructs a URL from memory. If
|
||||
the only thing the model has to emit is a small integer it was handed at
|
||||
retrieval time, there is nothing to reconstruct — a wrong id is detectable
|
||||
(it's not in the ledger) and a wrong URL is impossible (the model never types
|
||||
one; `render` does). This is the property Perplexity's product relies on, and
|
||||
it's why the ledger, not the prose, owns the URL.
|
||||
|
||||
Consequence: **register at retrieval, render mechanically.** Any workflow that
|
||||
lets the model type a URL into the Sources block gives the guarantee back.
|
||||
|
||||
## Cite while writing, not after
|
||||
|
||||
ALCE (arXiv:2305.14627) evaluates attribution for LLM answers and finds that
|
||||
generating citations during composition, from numbered retrieved snippets,
|
||||
produces materially better attribution than post-hoc citation insertion.
|
||||
Post-hoc attribution invites the model to find a source that plausibly matches
|
||||
a sentence it already wrote — which is exactly how a citation ends up
|
||||
supporting something the page doesn't say.
|
||||
|
||||
Hence: cite per supported sentence, in-line, as the sentence is written. Never
|
||||
a citation dump at the end of a paragraph or document.
|
||||
|
||||
## Verbatim quotes ground claims
|
||||
|
||||
WebGPT (arXiv:2112.09332) collects verbatim quotes at browse time and composes
|
||||
answers from them. The practical rule for this skill: when a claim carries a
|
||||
figure, date, name, or quantity, take it from the source's own words rather
|
||||
than paraphrasing from a summary of a summary. Each summarization hop is a
|
||||
chance for a number to drift.
|
||||
|
||||
## Formatting conventions
|
||||
|
||||
From Perplexity's leaked system prompts (jujumilk3/leaked-system-prompts) — the
|
||||
conventions are worth copying because they're the ones users have been trained
|
||||
to read:
|
||||
|
||||
- Marker directly after the terminal punctuation, no space: `water.[1]`
|
||||
- Each id in its own brackets: `[1][2]`, not `[1, 2]`
|
||||
- At most 3 ids per sentence — beyond that the citation stops identifying which
|
||||
source carries the claim
|
||||
- Never cite a source not actually consulted
|
||||
- Query classes that shouldn't be cited (translation, creative writing, casual
|
||||
chat) are exempted by instruction, not by a separate classifier
|
||||
|
||||
Perplexity forbids raw URLs in the answer because its UI renders source cards.
|
||||
Hermes has no such UI layer in chat or in a written file, so this skill renders
|
||||
the id → URL list explicitly instead.
|
||||
|
||||
## Related in-tree implementation
|
||||
|
||||
`tools/web_tools.py` grew an in-process version of this idea (a `url -> [n]`
|
||||
registry plus citation guidance attached to tool results) in PR #44833. That
|
||||
path grounds ad-hoc web answers automatically when it lands. This skill is the
|
||||
portable half: it works with any retrieval source (browser, curl, CLIs, local
|
||||
PDFs) and it persists the ledger to disk so multi-turn, multi-file, and
|
||||
multi-subagent work keeps stable ids. The two can coexist — the ledger is the
|
||||
source of truth for anything written to a file.
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Resolve HERMES_HOME for standalone skill scripts.
|
||||
|
||||
Skill scripts may run outside the Hermes process (system Python, nix env,
|
||||
CI) where ``hermes_constants`` is not importable. This module provides the
|
||||
same ``get_hermes_home()`` contract without requiring it on ``sys.path``.
|
||||
|
||||
When ``hermes_constants`` IS available it is used directly so profile
|
||||
resolution and any future enhancements are picked up automatically.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from hermes_constants import get_hermes_home as get_hermes_home
|
||||
except (ModuleNotFoundError, ImportError):
|
||||
|
||||
def get_hermes_home() -> Path:
|
||||
"""Return the Hermes home directory (default: ``~/.hermes``)."""
|
||||
val = os.environ.get("HERMES_HOME", "").strip()
|
||||
return Path(val) if val else Path.home() / ".hermes"
|
||||
@@ -0,0 +1,524 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Citation ledger for grounded answers and documents.
|
||||
|
||||
Owns the ``url -> [n]`` mapping used by the ``grounded-citations`` skill.
|
||||
Ids are assigned at retrieval time and never change, so a draft's ``[3]``
|
||||
always resolves to the same page. The model only ever emits integers the
|
||||
ledger handed it, which is what makes the citations verifiable.
|
||||
|
||||
Subcommands
|
||||
-----------
|
||||
reset start a clean ledger
|
||||
add URL [URL ...] register source(s), print their ids
|
||||
ingest FILE|- register every url found in JSON tool output
|
||||
list show the ledger
|
||||
render render a Sources block
|
||||
verify DRAFT check a draft's citations against the ledger
|
||||
|
||||
Ledger path resolution (first wins):
|
||||
--ledger PATH
|
||||
$HERMES_CITATION_LEDGER
|
||||
$HERMES_HOME/cache/citations/ledger.json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from _hermes_home import get_hermes_home # noqa: E402
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
|
||||
# A citation marker in prose: [12]. Markdown links ([text](url)) and
|
||||
# reference-style labels are excluded by requiring digits only and no
|
||||
# following "(" or ":".
|
||||
_CITE_RE = re.compile(r"\[(\d{1,4})\](?![(:])")
|
||||
_SOURCES_HEADER_RE = re.compile(r"^\s*(?:#{1,6}\s*)?(?:\*\*)?sources:?(?:\*\*)?\s*$", re.IGNORECASE)
|
||||
_SOURCE_LINE_RE = re.compile(r"^\s*\[(\d{1,4})\]\s*[-–:]?\s*(\S+)")
|
||||
_URL_IN_TEXT_RE = re.compile(r"https?://[^\s\"'<>)\]}]+")
|
||||
_FENCE_RE = re.compile(r"^\s*(?:```|~~~)")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ledger I/O
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def resolve_ledger_path(explicit: str | None = None) -> Path:
|
||||
if explicit:
|
||||
return Path(explicit).expanduser()
|
||||
env = os.environ.get("HERMES_CITATION_LEDGER", "").strip()
|
||||
if env:
|
||||
return Path(env).expanduser()
|
||||
return get_hermes_home() / "cache" / "citations" / "ledger.json"
|
||||
|
||||
|
||||
def normalize_url(url: str) -> str:
|
||||
"""Canonicalize a URL for ledger identity.
|
||||
|
||||
Strips the fragment and a trailing slash so ``/page``, ``/page/`` and
|
||||
``/page#section`` are one source. Query strings are significant and are
|
||||
kept — they usually select different content.
|
||||
"""
|
||||
u = (url or "").strip()
|
||||
if "#" in u:
|
||||
u = u.split("#", 1)[0]
|
||||
stripped = u.rstrip("/")
|
||||
return stripped or u
|
||||
|
||||
|
||||
def load_ledger(path: Path) -> dict[str, Any]:
|
||||
if not path.exists():
|
||||
return {"version": SCHEMA_VERSION, "sources": []}
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError) as exc:
|
||||
raise SystemExit(f"error: ledger at {path} is unreadable ({exc}); run `reset` to start over")
|
||||
if not isinstance(data, dict) or not isinstance(data.get("sources"), list):
|
||||
raise SystemExit(f"error: ledger at {path} has an unexpected shape; run `reset`")
|
||||
return data
|
||||
|
||||
|
||||
def save_ledger(path: Path, data: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(path.suffix + f".tmp{os.getpid()}")
|
||||
tmp.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
class _LedgerLock:
|
||||
"""Best-effort cross-process lock (O_EXCL lockfile, stdlib only).
|
||||
|
||||
Parallel subagents can share one ledger via --ledger; without a lock two
|
||||
concurrent ``add`` calls can assign the same id. Falls through after the
|
||||
timeout rather than blocking a task forever — a stale lock must never
|
||||
wedge the ledger.
|
||||
"""
|
||||
|
||||
def __init__(self, path: Path, timeout: float = 5.0) -> None:
|
||||
self.lock_path = path.with_suffix(path.suffix + ".lock")
|
||||
self.timeout = timeout
|
||||
self.fd: int | None = None
|
||||
|
||||
def __enter__(self) -> "_LedgerLock":
|
||||
self.lock_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
deadline = time.monotonic() + self.timeout
|
||||
while True:
|
||||
try:
|
||||
self.fd = os.open(str(self.lock_path), os.O_CREAT | os.O_EXCL | os.O_WRONLY)
|
||||
return self
|
||||
except FileExistsError:
|
||||
if time.monotonic() >= deadline:
|
||||
# Assume a stale lock from a crashed run.
|
||||
try:
|
||||
self.lock_path.unlink()
|
||||
except OSError:
|
||||
return self
|
||||
continue
|
||||
time.sleep(0.05)
|
||||
|
||||
def __exit__(self, *_exc: object) -> None:
|
||||
if self.fd is not None:
|
||||
try:
|
||||
os.close(self.fd)
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
self.lock_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Core operations
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _by_url(sources: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
|
||||
return {s["url"]: s for s in sources}
|
||||
|
||||
|
||||
def add_sources(
|
||||
path: Path,
|
||||
urls: Iterable[str],
|
||||
title: str | None = None,
|
||||
accessed: str | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Register urls, returning their ledger entries (existing or new)."""
|
||||
urls = [u for u in (str(u).strip() for u in urls) if u]
|
||||
if not urls:
|
||||
return []
|
||||
with _LedgerLock(path):
|
||||
data = load_ledger(path)
|
||||
sources = data["sources"]
|
||||
index = _by_url(sources)
|
||||
out: list[dict[str, Any]] = []
|
||||
changed = False
|
||||
for raw in urls:
|
||||
key = normalize_url(raw)
|
||||
existing = index.get(key)
|
||||
if existing is not None:
|
||||
if title and not existing.get("title"):
|
||||
existing["title"] = title
|
||||
changed = True
|
||||
out.append(existing)
|
||||
continue
|
||||
entry = {
|
||||
"id": len(sources) + 1,
|
||||
"url": key,
|
||||
"title": (title or "").strip(),
|
||||
"accessed": accessed or time.strftime("%Y-%m-%d"),
|
||||
}
|
||||
sources.append(entry)
|
||||
index[key] = entry
|
||||
out.append(entry)
|
||||
changed = True
|
||||
if changed:
|
||||
save_ledger(path, data)
|
||||
return out
|
||||
|
||||
|
||||
def urls_from_json(payload: Any) -> list[tuple[str, str]]:
|
||||
"""Walk arbitrary JSON tool output collecting (url, title) pairs.
|
||||
|
||||
Handles web_search (``data.web[]``), web_extract (``results[]``) and any
|
||||
other nesting, in document order, deduped.
|
||||
"""
|
||||
found: list[tuple[str, str]] = []
|
||||
seen: set[str] = set()
|
||||
|
||||
def walk(node: Any) -> None:
|
||||
if isinstance(node, dict):
|
||||
url = node.get("url") or node.get("link") or node.get("source_url")
|
||||
if isinstance(url, str) and url.startswith(("http://", "https://")):
|
||||
key = normalize_url(url)
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
raw_title = node.get("title") or node.get("name") or ""
|
||||
found.append((url, raw_title if isinstance(raw_title, str) else ""))
|
||||
for value in node.values():
|
||||
walk(value)
|
||||
elif isinstance(node, list):
|
||||
for item in node:
|
||||
walk(item)
|
||||
|
||||
walk(payload)
|
||||
return found
|
||||
|
||||
|
||||
def render_sources(
|
||||
sources: list[dict[str, Any]],
|
||||
style: str = "markdown",
|
||||
only: set[int] | None = None,
|
||||
) -> str:
|
||||
picked = [s for s in sources if only is None or s["id"] in only]
|
||||
picked.sort(key=lambda s: s["id"])
|
||||
if not picked:
|
||||
return ""
|
||||
lines: list[str] = []
|
||||
if style == "bibtex":
|
||||
for s in picked:
|
||||
key = f"source{s['id']}"
|
||||
title = s.get("title") or s["url"]
|
||||
lines.append(
|
||||
"@misc{%s,\n title = {%s},\n howpublished = {\\url{%s}},\n note = {Accessed %s}\n}"
|
||||
% (key, title, s["url"], s.get("accessed", ""))
|
||||
)
|
||||
return "\n".join(lines)
|
||||
if style == "footnotes":
|
||||
for s in picked:
|
||||
title = s.get("title")
|
||||
suffix = f" — {title}" if title else ""
|
||||
lines.append(f"[^{s['id']}]: {s['url']}{suffix}")
|
||||
return "\n".join(lines)
|
||||
header = "Sources:" if style == "plain" else "## Sources"
|
||||
lines.append(header)
|
||||
if style != "plain":
|
||||
lines.append("")
|
||||
for s in picked:
|
||||
title = s.get("title")
|
||||
suffix = f" — {title}" if title else ""
|
||||
lines.append(f"[{s['id']}] {s['url']}{suffix}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Verification
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _split_draft(text: str) -> tuple[str, dict[int, str]]:
|
||||
"""Split a draft into (prose, sources_block_map).
|
||||
|
||||
The sources block is everything after the last ``Sources`` header; its
|
||||
``[n] url`` lines are parsed out so they aren't counted as prose citations.
|
||||
Fenced code blocks are dropped from prose.
|
||||
"""
|
||||
lines = text.splitlines()
|
||||
header_idx = -1
|
||||
for i, line in enumerate(lines):
|
||||
if _SOURCES_HEADER_RE.match(line):
|
||||
header_idx = i
|
||||
listed: dict[int, str] = {}
|
||||
if header_idx >= 0:
|
||||
for line in lines[header_idx + 1:]:
|
||||
m = _SOURCE_LINE_RE.match(line)
|
||||
if m:
|
||||
url_match = _URL_IN_TEXT_RE.search(line)
|
||||
listed[int(m.group(1))] = url_match.group(0) if url_match else m.group(2)
|
||||
body_lines = lines[:header_idx]
|
||||
else:
|
||||
body_lines = lines
|
||||
prose: list[str] = []
|
||||
in_fence = False
|
||||
for line in body_lines:
|
||||
if _FENCE_RE.match(line):
|
||||
in_fence = not in_fence
|
||||
continue
|
||||
if not in_fence:
|
||||
prose.append(line)
|
||||
return "\n".join(prose), listed
|
||||
|
||||
|
||||
def _sentences(prose: str) -> list[str]:
|
||||
"""Rough sentence split over prose lines, skipping headings and tables."""
|
||||
out: list[str] = []
|
||||
for line in prose.splitlines():
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith("#") or stripped.startswith("|"):
|
||||
continue
|
||||
if stripped.startswith(">"):
|
||||
stripped = stripped.lstrip("> ").strip()
|
||||
for part in re.split(r"(?<=[.!?])\s+", stripped):
|
||||
part = part.strip()
|
||||
if len(part.split()) >= 4:
|
||||
out.append(part)
|
||||
return out
|
||||
|
||||
|
||||
def verify_draft(
|
||||
draft_path: Path,
|
||||
sources: list[dict[str, Any]],
|
||||
strict: bool = False,
|
||||
min_coverage: float | None = None,
|
||||
) -> tuple[int, list[str], list[str]]:
|
||||
"""Return (exit_code, errors, warnings)."""
|
||||
text = draft_path.read_text(encoding="utf-8")
|
||||
prose, listed = _split_draft(text)
|
||||
by_id = {s["id"]: s for s in sources}
|
||||
|
||||
errors: list[str] = []
|
||||
warnings: list[str] = []
|
||||
|
||||
cited = [int(m) for m in _CITE_RE.findall(prose)]
|
||||
cited_set = set(cited)
|
||||
|
||||
unknown = sorted(i for i in cited_set if i not in by_id)
|
||||
if unknown:
|
||||
errors.append(
|
||||
"citations not in the ledger (hallucinated or renumbered): "
|
||||
+ ", ".join(f"[{i}]" for i in unknown)
|
||||
)
|
||||
|
||||
if cited_set and not listed:
|
||||
errors.append("draft cites sources but has no `Sources:` block — run `render --cited-in`")
|
||||
|
||||
missing_from_block = sorted(cited_set - set(listed)) if listed else []
|
||||
if missing_from_block:
|
||||
errors.append(
|
||||
"cited but absent from the Sources block: "
|
||||
+ ", ".join(f"[{i}]" for i in missing_from_block)
|
||||
)
|
||||
|
||||
for sid, url in sorted(listed.items()):
|
||||
entry = by_id.get(sid)
|
||||
if entry is None:
|
||||
errors.append(f"Sources block lists [{sid}], which is not in the ledger")
|
||||
continue
|
||||
if normalize_url(url) != entry["url"]:
|
||||
errors.append(
|
||||
f"Sources block URL for [{sid}] does not match the ledger "
|
||||
f"(block: {url} / ledger: {entry['url']}) — re-run `render`"
|
||||
)
|
||||
|
||||
extra_in_block = sorted(set(listed) - cited_set)
|
||||
if extra_in_block:
|
||||
warnings.append(
|
||||
"listed in Sources but never cited inline: "
|
||||
+ ", ".join(f"[{i}]" for i in extra_in_block)
|
||||
)
|
||||
|
||||
registered_uncited = sorted(set(by_id) - cited_set)
|
||||
if registered_uncited:
|
||||
warnings.append(
|
||||
"registered in the ledger but not cited in this draft: "
|
||||
+ ", ".join(f"[{i}]" for i in registered_uncited)
|
||||
)
|
||||
|
||||
sentences = _sentences(prose)
|
||||
cited_sentences = [s for s in sentences if _CITE_RE.search(s)]
|
||||
coverage = (len(cited_sentences) / len(sentences)) if sentences else 0.0
|
||||
if min_coverage is not None and sentences and coverage < min_coverage:
|
||||
errors.append(
|
||||
f"citation coverage {coverage:.0%} is below the required {min_coverage:.0%} "
|
||||
f"({len(cited_sentences)}/{len(sentences)} sentences cited)"
|
||||
)
|
||||
|
||||
over_cited = [s for s in sentences if len(_CITE_RE.findall(s)) > 3]
|
||||
if over_cited:
|
||||
warnings.append(f"{len(over_cited)} sentence(s) carry more than 3 citations")
|
||||
|
||||
code = 1 if errors else (1 if (strict and warnings) else 0)
|
||||
stats = (
|
||||
f"{len(sentences)} prose sentence(s), {len(cited_sentences)} cited "
|
||||
f"({coverage:.0%}), {len(cited_set)} distinct source(s) cited, "
|
||||
f"{len(by_id)} in ledger"
|
||||
)
|
||||
warnings.insert(0, f"stats: {stats}")
|
||||
return code, errors, warnings
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _parse_only(spec: str | None) -> set[int] | None:
|
||||
if not spec:
|
||||
return None
|
||||
out: set[int] = set()
|
||||
for chunk in spec.replace(" ", "").split(","):
|
||||
if not chunk:
|
||||
continue
|
||||
if "-" in chunk:
|
||||
lo, _, hi = chunk.partition("-")
|
||||
out.update(range(int(lo), int(hi) + 1))
|
||||
else:
|
||||
out.add(int(chunk))
|
||||
return out
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="sources.py", description="Citation ledger for grounded answers and documents."
|
||||
)
|
||||
parser.add_argument("--ledger", help="ledger file path (overrides env / default)")
|
||||
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||
|
||||
sub.add_parser("reset", help="start a clean ledger")
|
||||
|
||||
p_add = sub.add_parser("add", help="register source url(s), print their ids")
|
||||
p_add.add_argument("urls", nargs="+")
|
||||
p_add.add_argument("--title", help="title for the source (single-url calls)")
|
||||
p_add.add_argument("--accessed", help="access date (default: today)")
|
||||
p_add.add_argument("--json", action="store_true", help="emit JSON instead of ids")
|
||||
|
||||
p_ing = sub.add_parser("ingest", help="register every url in JSON tool output")
|
||||
p_ing.add_argument("file", help="JSON file, or - for stdin")
|
||||
|
||||
p_list = sub.add_parser("list", help="show the ledger")
|
||||
p_list.add_argument("--json", action="store_true")
|
||||
|
||||
p_render = sub.add_parser("render", help="render a Sources block")
|
||||
p_render.add_argument(
|
||||
"--style", default="markdown", choices=["markdown", "plain", "footnotes", "bibtex"]
|
||||
)
|
||||
p_render.add_argument("--only", help="ids to include, e.g. 1,3,5-7")
|
||||
p_render.add_argument("--cited-in", help="include only ids cited in this draft file")
|
||||
|
||||
p_ver = sub.add_parser("verify", help="check a draft's citations against the ledger")
|
||||
p_ver.add_argument("draft")
|
||||
p_ver.add_argument("--strict", action="store_true", help="treat warnings as failures")
|
||||
p_ver.add_argument("--min-coverage", type=float, help="required cited-sentence share, e.g. 0.5")
|
||||
|
||||
args = parser.parse_args(argv)
|
||||
path = resolve_ledger_path(args.ledger)
|
||||
|
||||
if args.cmd == "reset":
|
||||
with _LedgerLock(path):
|
||||
save_ledger(path, {"version": SCHEMA_VERSION, "sources": []})
|
||||
print(f"ledger reset: {path}")
|
||||
return 0
|
||||
|
||||
if args.cmd == "add":
|
||||
title = args.title if len(args.urls) == 1 else None
|
||||
entries = add_sources(path, args.urls, title=title, accessed=args.accessed)
|
||||
if args.json:
|
||||
print(json.dumps(entries, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
for e in entries:
|
||||
print(f"[{e['id']}] {e['url']}")
|
||||
return 0
|
||||
|
||||
if args.cmd == "ingest":
|
||||
raw = sys.stdin.read() if args.file == "-" else Path(args.file).read_text(encoding="utf-8")
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
print(f"error: input is not valid JSON ({exc})", file=sys.stderr)
|
||||
return 2
|
||||
pairs = urls_from_json(payload)
|
||||
if not pairs:
|
||||
print("no urls found in input", file=sys.stderr)
|
||||
return 1
|
||||
for url, title in pairs:
|
||||
entry = add_sources(path, [url], title=title or None)[0]
|
||||
print(f"[{entry['id']}] {entry['url']}")
|
||||
return 0
|
||||
|
||||
data = load_ledger(path)
|
||||
sources = sorted(data["sources"], key=lambda s: s["id"])
|
||||
|
||||
if args.cmd == "list":
|
||||
if args.json:
|
||||
print(json.dumps(sources, indent=2, ensure_ascii=False))
|
||||
elif not sources:
|
||||
print(f"ledger is empty: {path}")
|
||||
else:
|
||||
for s in sources:
|
||||
title = f" {s['title']}" if s.get("title") else ""
|
||||
print(f"[{s['id']}] {s['url']}{title}")
|
||||
return 0
|
||||
|
||||
if args.cmd == "render":
|
||||
only = _parse_only(args.only)
|
||||
if args.cited_in:
|
||||
draft = Path(args.cited_in).read_text(encoding="utf-8")
|
||||
prose, _ = _split_draft(draft)
|
||||
cited = {int(m) for m in _CITE_RE.findall(prose)}
|
||||
only = cited if only is None else (only & cited)
|
||||
block = render_sources(sources, style=args.style, only=only)
|
||||
if not block:
|
||||
print("no sources to render", file=sys.stderr)
|
||||
return 1
|
||||
print(block)
|
||||
return 0
|
||||
|
||||
if args.cmd == "verify":
|
||||
draft_path = Path(args.draft)
|
||||
if not draft_path.is_file():
|
||||
print(f"error: no such draft: {draft_path}", file=sys.stderr)
|
||||
return 2
|
||||
code, errors, warnings = verify_draft(
|
||||
draft_path, sources, strict=args.strict, min_coverage=args.min_coverage
|
||||
)
|
||||
for w in warnings:
|
||||
print(f"warn: {w}")
|
||||
for e in errors:
|
||||
print(f"FAIL: {e}", file=sys.stderr)
|
||||
print("citations OK" if code == 0 else "verification failed")
|
||||
return code
|
||||
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,339 @@
|
||||
"""Tests for the grounded-citations bundled skill.
|
||||
|
||||
Covers the SKILL.md authoring standards (frontmatter shape, ≤60-char
|
||||
description) and the behavior of ``scripts/sources.py`` — the citation ledger
|
||||
that assigns stable ``url -> [n]`` ids, renders Sources blocks, and verifies a
|
||||
draft's citations. The verify path is the load-bearing piece: it is what
|
||||
catches a hallucinated or renumbered citation before delivery.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
SKILL_DIR = Path(__file__).resolve().parents[2] / "skills" / "research" / "grounded-citations"
|
||||
SCRIPT = SKILL_DIR / "scripts" / "sources.py"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def frontmatter() -> dict:
|
||||
src = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8")
|
||||
m = re.search(r"^---\n(.*?)\n---", src, re.DOTALL)
|
||||
assert m, "SKILL.md missing YAML frontmatter"
|
||||
return yaml.safe_load(m.group(1))
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def sources_mod():
|
||||
spec = importlib.util.spec_from_file_location("gc_sources", SCRIPT)
|
||||
assert spec is not None and spec.loader is not None
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ledger(tmp_path: Path) -> Path:
|
||||
return tmp_path / "ledger.json"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Authoring standards
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_skill_files_present() -> None:
|
||||
assert (SKILL_DIR / "SKILL.md").is_file()
|
||||
assert SCRIPT.is_file()
|
||||
assert (SKILL_DIR / "references" / "citation-formats.md").is_file()
|
||||
assert (SKILL_DIR / "references" / "grounding-rationale.md").is_file()
|
||||
|
||||
|
||||
def test_description_within_limit(frontmatter: dict) -> None:
|
||||
desc = frontmatter["description"]
|
||||
assert len(desc) <= 60, f"description is {len(desc)} chars (limit 60): {desc!r}"
|
||||
assert desc.endswith(".")
|
||||
|
||||
|
||||
def test_required_frontmatter_fields(frontmatter: dict) -> None:
|
||||
assert frontmatter["name"] == "grounded-citations"
|
||||
for field in ("version", "author", "license", "platforms"):
|
||||
assert frontmatter.get(field), f"missing frontmatter field: {field}"
|
||||
assert frontmatter["metadata"]["hermes"]["category"] == "research"
|
||||
|
||||
|
||||
def test_skill_body_has_modern_sections() -> None:
|
||||
body = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8")
|
||||
for heading in (
|
||||
"## When to Use",
|
||||
"## Prerequisites",
|
||||
"## How to Run",
|
||||
"## Quick Reference",
|
||||
"## Procedure",
|
||||
"## Pitfalls",
|
||||
"## Verification",
|
||||
):
|
||||
assert heading in body, f"SKILL.md missing section: {heading}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ledger identity
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_ids_are_stable_and_sequential(sources_mod, ledger: Path) -> None:
|
||||
first = sources_mod.add_sources(ledger, ["https://a.example"])
|
||||
second = sources_mod.add_sources(ledger, ["https://b.example"])
|
||||
assert (first[0]["id"], second[0]["id"]) == (1, 2)
|
||||
again = sources_mod.add_sources(ledger, ["https://a.example"])
|
||||
assert again[0]["id"] == 1
|
||||
|
||||
|
||||
def test_url_normalization_collapses_fragment_and_trailing_slash(sources_mod, ledger: Path) -> None:
|
||||
base = sources_mod.add_sources(ledger, ["https://x.example/page"])[0]["id"]
|
||||
for variant in ("https://x.example/page/", "https://x.example/page#part"):
|
||||
assert sources_mod.add_sources(ledger, [variant])[0]["id"] == base
|
||||
|
||||
|
||||
def test_query_string_is_significant(sources_mod, ledger: Path) -> None:
|
||||
a = sources_mod.add_sources(ledger, ["https://x.example/s?q=1"])[0]["id"]
|
||||
b = sources_mod.add_sources(ledger, ["https://x.example/s?q=2"])[0]["id"]
|
||||
assert a != b
|
||||
|
||||
|
||||
def test_title_backfills_without_changing_id(sources_mod, ledger: Path) -> None:
|
||||
first = sources_mod.add_sources(ledger, ["https://t.example"])[0]
|
||||
assert first["title"] == ""
|
||||
second = sources_mod.add_sources(ledger, ["https://t.example"], title="Later title")[0]
|
||||
assert (second["id"], second["title"]) == (first["id"], "Later title")
|
||||
|
||||
|
||||
def test_ingest_walks_search_and_extract_payloads(sources_mod) -> None:
|
||||
payload = {
|
||||
"data": {"web": [{"title": "One", "url": "https://n.example/1"}]},
|
||||
"results": [
|
||||
{"url": "https://n.example/1", "title": "One again"},
|
||||
{"url": "https://n.example/2", "title": "Two"},
|
||||
],
|
||||
}
|
||||
pairs = sources_mod.urls_from_json(payload)
|
||||
assert [u for u, _ in pairs] == ["https://n.example/1", "https://n.example/2"]
|
||||
|
||||
|
||||
def test_ingest_ignores_non_http_values(sources_mod) -> None:
|
||||
payload = {"url": "file:///etc/passwd", "nested": {"link": "mailto:a@b.c"}}
|
||||
assert sources_mod.urls_from_json(payload) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rendering
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed(sources_mod, ledger: Path) -> list[dict]:
|
||||
sources_mod.add_sources(ledger, ["https://a.example"], title="Alpha")
|
||||
sources_mod.add_sources(ledger, ["https://b.example"])
|
||||
sources_mod.add_sources(ledger, ["https://c.example"], title="Gamma")
|
||||
return json.loads(ledger.read_text(encoding="utf-8"))["sources"]
|
||||
|
||||
|
||||
def test_render_markdown_lists_ids_and_urls(sources_mod, ledger: Path) -> None:
|
||||
block = sources_mod.render_sources(_seed(sources_mod, ledger))
|
||||
assert block.startswith("## Sources")
|
||||
assert "[1] https://a.example — Alpha" in block
|
||||
assert "[2] https://b.example" in block
|
||||
|
||||
|
||||
def test_render_only_subset_and_ordering(sources_mod, ledger: Path) -> None:
|
||||
block = sources_mod.render_sources(_seed(sources_mod, ledger), style="plain", only={3, 1})
|
||||
lines = [ln for ln in block.splitlines() if ln.startswith("[")]
|
||||
assert lines[0].startswith("[1]") and lines[1].startswith("[3]")
|
||||
assert not any(ln.startswith("[2]") for ln in lines)
|
||||
|
||||
|
||||
def test_render_bibtex_keys_match_ids(sources_mod, ledger: Path) -> None:
|
||||
block = sources_mod.render_sources(_seed(sources_mod, ledger), style="bibtex", only={1})
|
||||
assert "@misc{source1," in block
|
||||
assert r"\url{https://a.example}" in block
|
||||
|
||||
|
||||
def test_render_empty_selection_is_empty_string(sources_mod, ledger: Path) -> None:
|
||||
assert sources_mod.render_sources(_seed(sources_mod, ledger), only=set()) == ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Verification — the guarantee
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _verify(sources_mod, ledger: Path, tmp_path: Path, text: str, **kw):
|
||||
draft = tmp_path / "draft.md"
|
||||
draft.write_text(text, encoding="utf-8")
|
||||
sources = json.loads(ledger.read_text(encoding="utf-8"))["sources"]
|
||||
return sources_mod.verify_draft(draft, sources, **kw)
|
||||
|
||||
|
||||
def test_well_formed_draft_passes(sources_mod, ledger: Path, tmp_path: Path) -> None:
|
||||
_seed(sources_mod, ledger)
|
||||
text = (
|
||||
"Ice is less dense than liquid water and floats.[1][2]\n\n"
|
||||
"Sources:\n[1] https://a.example\n[2] https://b.example\n"
|
||||
)
|
||||
code, errors, _ = _verify(sources_mod, ledger, tmp_path, text)
|
||||
assert (code, errors) == (0, [])
|
||||
|
||||
|
||||
def test_unknown_citation_id_fails(sources_mod, ledger: Path, tmp_path: Path) -> None:
|
||||
_seed(sources_mod, ledger)
|
||||
text = "A claim with an invented source id here.[42]\n\nSources:\n[42] https://fake.example\n"
|
||||
code, errors, _ = _verify(sources_mod, ledger, tmp_path, text)
|
||||
assert code == 1
|
||||
# Must be flagged as an inline citation the ledger never issued — not merely
|
||||
# as a stray Sources-block line, which is a separate (weaker) error.
|
||||
assert any("hallucinated or renumbered" in e for e in errors), errors
|
||||
|
||||
|
||||
def test_sources_block_url_must_match_ledger(sources_mod, ledger: Path, tmp_path: Path) -> None:
|
||||
_seed(sources_mod, ledger)
|
||||
text = "A real claim carrying a real id.[1]\n\nSources:\n[1] https://wrong.example\n"
|
||||
code, errors, _ = _verify(sources_mod, ledger, tmp_path, text)
|
||||
assert code == 1
|
||||
assert any("does not match the ledger" in e for e in errors)
|
||||
|
||||
|
||||
def test_missing_sources_block_fails(sources_mod, ledger: Path, tmp_path: Path) -> None:
|
||||
_seed(sources_mod, ledger)
|
||||
code, errors, _ = _verify(sources_mod, ledger, tmp_path, "A real claim carrying an id.[1]\n")
|
||||
assert code == 1
|
||||
assert any("no `Sources:` block" in e for e in errors)
|
||||
|
||||
|
||||
def test_cited_but_absent_from_block_fails(sources_mod, ledger: Path, tmp_path: Path) -> None:
|
||||
_seed(sources_mod, ledger)
|
||||
text = (
|
||||
"First claim about the topic at hand.[1]\n"
|
||||
"Second claim about the topic at hand.[2]\n\n"
|
||||
"Sources:\n[1] https://a.example\n"
|
||||
)
|
||||
code, errors, _ = _verify(sources_mod, ledger, tmp_path, text)
|
||||
assert code == 1
|
||||
assert any("absent from the Sources block" in e for e in errors)
|
||||
|
||||
|
||||
def test_brackets_inside_code_fences_are_not_citations(sources_mod, ledger: Path, tmp_path: Path) -> None:
|
||||
_seed(sources_mod, ledger)
|
||||
text = "Prose with no external claims in it.\n\n```python\nvalue = arr[42]\n```\n"
|
||||
code, errors, _ = _verify(sources_mod, ledger, tmp_path, text)
|
||||
assert (code, errors) == (0, [])
|
||||
|
||||
|
||||
def test_markdown_links_are_not_citations(sources_mod, ledger: Path, tmp_path: Path) -> None:
|
||||
_seed(sources_mod, ledger)
|
||||
text = "See the [docs](https://x.example) for the full option list.\n"
|
||||
code, errors, _ = _verify(sources_mod, ledger, tmp_path, text)
|
||||
assert (code, errors) == (0, [])
|
||||
|
||||
|
||||
def test_min_coverage_gate(sources_mod, ledger: Path, tmp_path: Path) -> None:
|
||||
_seed(sources_mod, ledger)
|
||||
text = (
|
||||
"A cited claim about the subject matter.[1]\n"
|
||||
"An uncited claim about the subject matter.\n"
|
||||
"Another uncited claim about the subject matter.\n"
|
||||
"A third uncited claim about the subject matter.\n\n"
|
||||
"Sources:\n[1] https://a.example\n"
|
||||
)
|
||||
ok_code, _, _ = _verify(sources_mod, ledger, tmp_path, text)
|
||||
assert ok_code == 0
|
||||
code, errors, _ = _verify(sources_mod, ledger, tmp_path, text, min_coverage=0.5)
|
||||
assert code == 1
|
||||
assert any("coverage" in e for e in errors)
|
||||
|
||||
|
||||
def test_over_citation_warns_without_failing(sources_mod, ledger: Path, tmp_path: Path) -> None:
|
||||
sources_mod.add_sources(
|
||||
ledger, [f"https://s{i}.example" for i in range(1, 5)]
|
||||
)
|
||||
text = (
|
||||
"One sentence leaning on far too many sources at once.[1][2][3][4]\n\n"
|
||||
"Sources:\n"
|
||||
+ "".join(f"[{i}] https://s{i}.example\n" for i in range(1, 5))
|
||||
)
|
||||
code, errors, warnings = _verify(sources_mod, ledger, tmp_path, text)
|
||||
assert (code, errors) == (0, [])
|
||||
assert any("more than 3 citations" in w for w in warnings)
|
||||
|
||||
|
||||
def test_strict_mode_promotes_warnings_to_failure(sources_mod, ledger: Path, tmp_path: Path) -> None:
|
||||
_seed(sources_mod, ledger)
|
||||
text = "A cited claim about the subject.[1]\n\nSources:\n[1] https://a.example\n"
|
||||
assert _verify(sources_mod, ledger, tmp_path, text)[0] == 0
|
||||
assert _verify(sources_mod, ledger, tmp_path, text, strict=True)[0] == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI surface
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cli_add_render_verify_roundtrip(sources_mod, tmp_path: Path, capsys) -> None:
|
||||
ledger = tmp_path / "cli.json"
|
||||
args = ["--ledger", str(ledger)]
|
||||
assert sources_mod.main(args + ["add", "https://a.example", "https://b.example"]) == 0
|
||||
assert "[1] https://a.example" in capsys.readouterr().out
|
||||
|
||||
draft = tmp_path / "d.md"
|
||||
draft.write_text("Only the first source is used here.[1]\n", encoding="utf-8")
|
||||
assert sources_mod.main(args + ["render", "--cited-in", str(draft)]) == 0
|
||||
block = capsys.readouterr().out
|
||||
assert "[1] https://a.example" in block and "[2]" not in block
|
||||
|
||||
with draft.open("a", encoding="utf-8") as fh:
|
||||
fh.write("\n" + block)
|
||||
assert sources_mod.main(args + ["verify", str(draft)]) == 0
|
||||
|
||||
|
||||
def test_cli_reset_empties_the_ledger(sources_mod, tmp_path: Path, capsys) -> None:
|
||||
ledger = tmp_path / "r.json"
|
||||
args = ["--ledger", str(ledger)]
|
||||
sources_mod.main(args + ["add", "https://a.example"])
|
||||
capsys.readouterr()
|
||||
assert sources_mod.main(args + ["reset"]) == 0
|
||||
capsys.readouterr()
|
||||
assert sources_mod.main(args + ["add", "https://z.example"]) == 0
|
||||
assert "[1] https://z.example" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_cli_verify_missing_draft_returns_2(sources_mod, tmp_path: Path) -> None:
|
||||
ledger = tmp_path / "m.json"
|
||||
code = sources_mod.main(["--ledger", str(ledger), "verify", str(tmp_path / "nope.md")])
|
||||
assert code == 2
|
||||
|
||||
|
||||
def test_cli_ledger_path_prefers_flag_over_env(sources_mod, tmp_path: Path, monkeypatch) -> None:
|
||||
monkeypatch.setenv("HERMES_CITATION_LEDGER", str(tmp_path / "env.json"))
|
||||
flagged = tmp_path / "flag.json"
|
||||
assert sources_mod.resolve_ledger_path(str(flagged)) == flagged
|
||||
assert sources_mod.resolve_ledger_path(None) == tmp_path / "env.json"
|
||||
|
||||
|
||||
def test_cli_ledger_path_defaults_under_hermes_home(sources_mod, tmp_path: Path, monkeypatch) -> None:
|
||||
monkeypatch.delenv("HERMES_CITATION_LEDGER", raising=False)
|
||||
monkeypatch.setenv("HERMES_HOME", str(tmp_path / ".hermes"))
|
||||
path = sources_mod.resolve_ledger_path(None)
|
||||
assert path.parts[-3:] == ("cache", "citations", "ledger.json")
|
||||
assert str(tmp_path) in str(path)
|
||||
|
||||
|
||||
def test_corrupt_ledger_raises_actionable_error(sources_mod, tmp_path: Path) -> None:
|
||||
bad = tmp_path / "bad.json"
|
||||
bad.write_text("not json", encoding="utf-8")
|
||||
with pytest.raises(SystemExit) as exc:
|
||||
sources_mod.load_ledger(bad)
|
||||
assert "reset" in str(exc.value)
|
||||
@@ -115,6 +115,7 @@ If a skill is missing from this list but present in the repo, the catalog is reg
|
||||
|-------|-------------|------|
|
||||
| [`arxiv`](/docs/user-guide/skills/bundled/research/research-arxiv) | Search arXiv papers by keyword, author, category, or ID. | `research/arxiv` |
|
||||
| [`blogwatcher`](/docs/user-guide/skills/bundled/research/research-blogwatcher) | Monitor blogs and RSS/Atom feeds via blogwatcher-cli tool. | `research/blogwatcher` |
|
||||
| [`grounded-citations`](/docs/user-guide/skills/bundled/research/research-grounded-citations) | Ground answers and documents in cited, verifiable sources. | `research/grounded-citations` |
|
||||
| [`llm-wiki`](/docs/user-guide/skills/bundled/research/research-llm-wiki) | Karpathy's LLM Wiki: build/query interlinked markdown KB. | `research/llm-wiki` |
|
||||
| [`polymarket`](/docs/user-guide/skills/bundled/research/research-polymarket) | Query Polymarket: markets, prices, orderbooks, history. | `research/polymarket` |
|
||||
| [`research-paper-writing`](/docs/user-guide/skills/bundled/research/research-research-paper-writing) | Write ML papers for NeurIPS/ICML/ICLR: design→submit. | `research/research-paper-writing` |
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
---
|
||||
title: "Grounded Citations — Ground answers and documents in cited, verifiable sources"
|
||||
sidebar_label: "Grounded Citations"
|
||||
description: "Ground answers and documents in cited, verifiable sources"
|
||||
---
|
||||
|
||||
{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
|
||||
|
||||
# Grounded Citations
|
||||
|
||||
Ground answers and documents in cited, verifiable sources.
|
||||
|
||||
## Skill metadata
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Source | Bundled (installed by default) |
|
||||
| Path | `skills/research/grounded-citations` |
|
||||
| Version | `1.0.0` |
|
||||
| Author | Hermes Agent + Teknium |
|
||||
| License | MIT |
|
||||
| Platforms | linux, macos, windows |
|
||||
| Tags | `Research`, `Citations`, `Grounding`, `Sources`, `Web`, `Reports` |
|
||||
| Related skills | [`research-paper-writing`](/docs/user-guide/skills/bundled/research/research-research-paper-writing), [`arxiv`](/docs/user-guide/skills/bundled/research/research-arxiv), [`ocr-and-documents`](/docs/user-guide/skills/bundled/productivity/productivity-ocr-and-documents) |
|
||||
|
||||
## Reference: full SKILL.md
|
||||
|
||||
:::info
|
||||
The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
|
||||
:::
|
||||
|
||||
# Grounded Citations
|
||||
|
||||
Every claim taken from an outside source gets an inline numbered citation and a
|
||||
`Sources:` list, Perplexity-style. A ledger script owns the `url → [n]` mapping
|
||||
so the numbers and URLs come from retrieval, never from memory — the model only
|
||||
ever emits small integers it was handed.
|
||||
|
||||
This skill covers answers in chat, written documents (markdown, PDF, docx,
|
||||
slides), and research reports. It does not cover academic BibTeX pipelines —
|
||||
for conference papers use the `research-paper-writing` skill, which this skill
|
||||
feeds (see `references/citation-formats.md`).
|
||||
|
||||
## When to Use
|
||||
|
||||
Use whenever an answer or artifact rests on information you fetched rather than
|
||||
knew:
|
||||
|
||||
- Research, comparisons, news summaries, "what is the current state of X"
|
||||
- Any deliverable you write to disk that quotes, paraphrases, or reports
|
||||
outside facts — reports, briefs, docs, decks, wiki pages
|
||||
- Fact-finding where the user will want to check your work
|
||||
- Multi-source synthesis where conflicting sources must be attributed
|
||||
|
||||
Skip inline citations when the retrieval is incidental to another task — a
|
||||
quick syntax/version lookup mid-coding, casual conversation, creative writing.
|
||||
Mention a URL only if the user would plausibly want the link.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
None beyond the standard toolset. `scripts/sources.py` is stdlib-only Python 3.
|
||||
Retrieval comes from whatever is configured: `web_search`, `web_extract`,
|
||||
`browser_navigate`, or `terminal` (curl, CLIs).
|
||||
|
||||
Ledger location: `$HERMES_HOME/cache/citations/ledger.json` (profile-aware).
|
||||
Override per task with `--ledger <path>` or `HERMES_CITATION_LEDGER`.
|
||||
|
||||
## How to Run
|
||||
|
||||
```bash
|
||||
S=~/.hermes/skills/research/grounded-citations/scripts/sources.py
|
||||
|
||||
python3 "$S" reset # start a clean ledger
|
||||
python3 "$S" add https://example.com/a --title "A" # prints: [1]
|
||||
python3 "$S" add https://example.com/b --title "B" # prints: [2]
|
||||
python3 "$S" list # ledger table
|
||||
python3 "$S" render # Sources: block
|
||||
python3 "$S" verify draft.md # catch bad citations
|
||||
```
|
||||
|
||||
`add` is idempotent and URL-normalized: the same page always returns the same
|
||||
id within a ledger, so ids stay stable across many search/extract rounds.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Action | Command |
|
||||
|---|---|
|
||||
| Fresh ledger for a new task | `sources.py reset` |
|
||||
| Register a source, get its id | `sources.py add <url> [--title T]` |
|
||||
| Register several at once | `sources.py add <url1> <url2> ...` |
|
||||
| Register from JSON tool output | `sources.py ingest results.json` |
|
||||
| Show ledger | `sources.py list [--json]` |
|
||||
| Render the Sources block | `sources.py render [--style markdown\|plain\|footnotes\|bibtex] [--only 1,3]` |
|
||||
| Render only what a draft cites | `sources.py render --cited-in draft.md` |
|
||||
| Check a draft's citations | `sources.py verify draft.md [--strict] [--min-coverage 0.6]` |
|
||||
|
||||
## Procedure
|
||||
|
||||
① **Reset the ledger** at the start of a task that will produce a grounded
|
||||
answer or document. Skip the reset when continuing work whose ids are already
|
||||
in a draft — reusing the ledger keeps the numbering stable.
|
||||
|
||||
② **Register every source at retrieval time.** After each `web_search` /
|
||||
`web_extract` / `browser_navigate` / fetch, pass the URLs to `sources.py add`
|
||||
(or pipe the raw JSON through `sources.py ingest`). Do this *before* writing
|
||||
prose. Registering later, from memory, is the failure mode this skill exists to
|
||||
prevent.
|
||||
|
||||
③ **Write cite-while-drafting.** Place the bracketed id(s) immediately after
|
||||
each sentence the source supports:
|
||||
|
||||
```
|
||||
Ice floats because it is less dense than liquid water.[1][2]
|
||||
```
|
||||
|
||||
- No space before the bracket; each id in its own brackets.
|
||||
- Max 3 ids per sentence. Cite per sentence, not one dump at the end.
|
||||
- Only ids the ledger returned. Never invent an id or a URL.
|
||||
- Claims from your own knowledge get no citation.
|
||||
- Conflicting sources: present both readings, each with its own id.
|
||||
- Quote exact figures, dates, and names as the source states them; flag gaps
|
||||
explicitly ("no source found for X") instead of smoothing them over.
|
||||
|
||||
④ **Append the Sources block** with `sources.py render --cited-in <draft>` so
|
||||
the id → URL mapping is generated mechanically from the ledger, not retyped.
|
||||
For non-markdown targets pick the matching `--style` and follow
|
||||
`references/citation-formats.md` for placement (footnotes in docx, endnotes in
|
||||
PDF/LaTeX, a Sources slide in decks, per-page source lists in wiki output).
|
||||
|
||||
⑤ **Verify before delivering** — `sources.py verify <draft>` exits non-zero on
|
||||
unknown ids, on a Sources block that disagrees with the ledger, or (with
|
||||
`--min-coverage`) on prose that is too thinly cited. Fix and re-run.
|
||||
|
||||
⑥ **Chat answers** follow the same steps with the draft in your reply: register
|
||||
sources, cite inline, end with the rendered `Sources:` list. For a short answer
|
||||
you may render the block from `sources.py render --only <ids>` instead of
|
||||
writing to a file.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Registering after writing.** The ledger must be populated from tool output,
|
||||
not reconstructed from the draft — that reintroduces exactly the hallucinated
|
||||
-URL risk the numbering removes.
|
||||
- **Renumbering mid-task.** Never hand-edit ids in a draft. Ids are ledger
|
||||
identities; if a draft cites `[4]`, `[4]` must stay that source. Run `reset`
|
||||
only between tasks.
|
||||
- **Retyping URLs into the Sources block.** Always `render`. A hand-typed URL
|
||||
is an unverified claim.
|
||||
- **Citing a search snippet as if you read the page.** A `web_search`
|
||||
description supports only what it literally says. Cite the extracted page
|
||||
when the claim needs the body — `web_extract` it first.
|
||||
- **Over-citing.** Three ids on a sentence is the ceiling; a citation on every
|
||||
clause makes text unreadable and hides which source carries the load.
|
||||
- **Citing the ledger in code/config artifacts.** Source comments belong in
|
||||
prose deliverables and doc headers, not inside generated code.
|
||||
- **Parallel subagents.** Each subagent has its own working directory; point
|
||||
them all at one ledger with `--ledger` (or `HERMES_CITATION_LEDGER`) if their
|
||||
outputs get merged, otherwise their ids will collide.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
python3 "$S" verify report.md --strict --min-coverage 0.5
|
||||
```
|
||||
|
||||
Green means: every `[n]` in the draft exists in the ledger, the Sources block
|
||||
lists exactly the cited ids with the ledger's URLs, and the cited share of
|
||||
source-bearing sentences meets the threshold. Read the warnings even when the
|
||||
exit code is 0 — uncited registered sources usually mean a claim lost its
|
||||
attribution during editing.
|
||||
@@ -276,6 +276,7 @@ const sidebars: SidebarsConfig = {
|
||||
items: [
|
||||
'user-guide/skills/bundled/research/research-arxiv',
|
||||
'user-guide/skills/bundled/research/research-blogwatcher',
|
||||
'user-guide/skills/bundled/research/research-grounded-citations',
|
||||
'user-guide/skills/bundled/research/research-llm-wiki',
|
||||
'user-guide/skills/bundled/research/research-polymarket',
|
||||
'user-guide/skills/bundled/research/research-research-paper-writing',
|
||||
|
||||
Reference in New Issue
Block a user