Commit Graph

5 Commits

Author SHA1 Message Date
Teknium c49fa88b80 refactor(skills): shipped-set slim — 15 to optional, github 6-way merge, pdf absorbs OCR, channel-gated teams pipeline (index −26%) (#98539)
* refactor(skills): shipped-set slim — 15 skills to optional, github six-way merge, pdf absorbs OCR+nano-pdf, channel-gated teams pipeline

Maintainer-directed shipped-skills curation (skills index 1,900 -> ~1,400
tok/call on desktop; every session pays the index, so this is a per-call
diet on all installs):

- optional-skills moves (installable via skills hub, history preserved):
  creative comfyui/ascii-art/excalidraw/pretext/sketch/touchdesigner-mcp;
  ALL of mlops (huggingface-hub, llama-cpp, serving-llms-vllm,
  weights-and-biases, evaluating-llms-harness — subcategory structure
  kept); research-paper-writing (55 supporting files, 17.3K-tok load);
  openhue; blogwatcher (first taught the cronjob monitor-field watch
  pattern + web_extract instead of pre-cron manual workflows)
- DELETED session-librarian (Aug-12 'inspired by Perplexity Computer'
  port, never maintainer-intended; session_search covers discovery)
- github: six skills (auth, issues, pr-workflow, issue-to-pr,
  code-review, repo-management) merged into ONE software-development/
  github skill — routing body + complete per-workflow references;
  benbarclay authorship credited; codebase-inspection rides along;
  discipline pins from test_github_issue_to_pr_skill.py preserved
  against the reference body in the new test_github_skill.py
- pdf absorbs ocr-and-documents + nano-pdf as references/ + scripts
  (extract_pymupdf, extract_marker converted to the argparse house
  standard its contract test enforces)
- NEW session_platforms frontmatter gate (metadata.hermes): hides a
  skill from the index on gateway channels it is not for; fail-open on
  unknown platform; teams-meeting-pipeline gated to [teams, cron]
- blocked-page-recovery: research -> new web category; trigger-first
  description ('Use when a fetch fails: 403/429, paywall, WAF, bot
  wall.') so the model actually reaches for it on blocked fetches
- docs regenerated via generate-skill-docs.py (195 pages); related_skills
  swept repo-wide; tests: 1672 passed (2 openclaw failures pre-existing
  on clean main, Windows-local)

* chore: ignore .skills_prompt_snapshot.json (local index cache, accidentally committed)
2026-08-30 04:53:39 -07:00
Teknium fad88cf130 feat: extend clean-room office skills toward full parity
Same clean-room discipline as the initial rewrite (isolated subagents,
functional specs only, predecessor content banned including via git
history; transcripts retained). All additions test-proven.

docx (13->29 tests):
- docx_revisions.py: tracked changes list/accept/reject (all or by id),
  incl. tables and headers/footers, via direct oxml manipulation
- docx_comments.py: list/add/delete comments (native python-docx >=1.2
  API with XML fallback), anchored-text extraction
- docx_validate.py: package health check (rels, images, styles, CRC)
  with JSON severity report — explicitly not XSD validation
- docx_edit.py: run normalization; TOC + PAGE/NUMPAGES field insertion

xlsx (5->12 tests):
- xlsx_restructure.py: reference-aware insert/delete rows/cols —
  rewrites formulas on all sheets (absolute refs, ranges, cross-sheet,
  quoted names), shifts merges/autofilter/freeze/validation/CF ranges,
  tables, defined names; JSON report incl. honest not_shifted list
- native Excel tables, named ranges, hyperlinks, cell notes,
  sheet protection (documented as strippable, not security)
- xlsx_recalc.py: headless LibreOffice recalc with graceful degrade

powerpoint (11->21 tests):
- pptx_render.py: all slides -> PNGs (soffice + pdftoppm/pdftocairo),
  wired to vision_analyze review loop in SKILL.md
- run-merge normalize before replace (identical-format splits lossless)
- surgical chart ops (series/category/title) wrapping replace_data
- slide duplication with rel remap (clean refusal on chart slides)
- backgrounds, hyperlinks, slide numbers, footers, notes editing

pdf (8->21 tests):
- pdf_make_form.py: JSON spec -> AcroForm (text/checkbox/radio/dropdown)
- pdf_form_layout.py: pre-build layout lint (bounds/overlap/pairing)
  + rendered box overlay for vision_analyze review
- pdf_page_image.py + shared _raster.py: pypdfium2 -> pdftoppm chain,
  graceful degrade; connected to scanned-PDF triage flow
- pdf_stamp.py: text/image stamps at coordinates (rotation/opacity)
- pdf_meta.py: DocInfo metadata + attachments round-trip

Gates re-verified independently: 83 skill tests green under LC_ALL=C,
repo invariant suite 29/29, SkillEvaluator pii+unicode+lint 3/3 x4.
2026-08-08 10:46:20 -07:00
Teknium 51570f4da7 feat: replace Anthropic office document skills with clean-room MIT implementations
The bundled docx, xlsx, powerpoint, and pdf skills were adapted from
Anthropic's document skills and carried their proprietary LICENSE.txt
(no derivatives, no redistribution). Flagged as critical license
findings by the SkillEvaluator Tier 1 scan of our skill tree.

This replaces all four with clean-room rewrites:

- Authored from scratch against library knowledge only (python-docx,
  openpyxl, python-pptx, pypdf/reportlab/pdfplumber — all MIT/BSD) by
  isolated subagents given functional specs, with an explicit
  prohibition on reading the prior skill content or anthropics/skills;
  session transcripts retained as provenance evidence.
- MIT licensed (LICENSE file per skill), author: Nous Research.
- Each skill: SKILL.md to house standards + argparse helper scripts
  with UTF-8-explicit I/O + its own e2e pytest suite (fixtures built
  on the fly, non-ASCII round-trips run under LC_ALL=C).
- All four pass SkillEvaluator Tier 1 pii+unicode+lint 3/3.

tests/skills/test_office_document_skills.py rewritten against the new
contracts: MIT/no-Anthropic-text invariants, scripts documented in
SKILL.md, argparse CLI shape, and a no-locale-default-open() check
(which caught and fixed a real gap: pdfplumber text reads are fine,
but the invariant scan now guards every future script).

Docs pages regenerated for the four skills (scoped; unrelated
generator drift excluded).

Honest capability deltas vs the old versions are documented per
SKILL.md (e.g. tracked-changes accept/reject and OOXML XSD validation
are not reimplemented; form flattening limits stated).
2026-08-08 10:46:20 -07:00
solyanviktor-star adecb0d1a9 fix(skills): read OOXML parts as bytes and form JSON as UTF-8 in office skill scripts
The bundled office skills (#68595) read user documents and agent-authored
payloads with the locale-default codec:

- docx/powerpoint validators/base.py opened OOXML part XML in text mode
  before handing it to lxml. On Windows (cp1251/GBK) the bytes decode to
  mojibake that lxml then parses, so validation runs against silently
  corrupted document text; on locales where the UTF-8 bytes don't decode
  the validator crashes with UnicodeDecodeError instead of validating.
  Opening as bytes lets lxml honor the encoding declared in the XML prolog.

- The pdf form scripts (fill_fillable_fields, fill_pdf_form_with_annotations,
  create_validation_image, check_bounding_boxes) read the fields JSON the
  agent authors — UTF-8 by construction — with the locale codec, so
  non-ASCII form values (any Cyrillic/CJK/accented input) either crash or
  get written into the user's PDF as mojibake. The json.dump writers use
  ensure_ascii=True and were already safe; only the readers needed pinning.

Adds a contract test asserting every document/payload reader is
locale-independent, plus a live regression test that runs
check_bounding_boxes.py on a non-ASCII fields.json under a forced
non-UTF-8 locale — it fails without the fix on both POSIX (C locale)
and Windows (cp1251 chokes on the 0x98 byte of U+2018).
2026-07-24 17:10:39 -07:00
Teknium afb7bf6a5a feat(skills): bundle docx, xlsx, and pdf office skills; refresh powerpoint (#68595)
Non-technical users asking for Word docs, spreadsheets, or PDF work had
no bundled skill coverage — docx/xlsx creation required discovering and
installing hub skills, and PDF manipulation had no skill at all beyond
OCR extraction and nano-pdf edits.

- skills/productivity/docx: create (docx-js), edit (unzip -> XML -> zip),
  tracked changes, comments, validation. Adapted from anthropics/skills.
- skills/productivity/xlsx: openpyxl creation/editing, mandatory
  LibreOffice recalc gate, formula-compatibility rules, financial-model
  conventions. Points at optional excel-author for finance-grade work.
- skills/productivity/pdf: merge/split/rotate/watermark/encrypt, form
  filling (AcroForm + flat overlay scripts), text/table extraction,
  reportlab creation, forms.md + reference.md companions.
- skills/productivity/powerpoint: synced to current upstream pptx skill —
  richer pptxgenjs corruption footguns, template workflow, validate.py +
  validators + thumbnail.py, font-substitution QA guidance; drops the
  stale pack.py/editing.md/pptxgenjs.md workflow files.
- Cross-linked ocr-and-documents, nano-pdf, excel-author via
  related_skills so each office skill routes to its siblings.
- deliverable-mode docs mention the new skills; regenerated per-skill
  docs pages, catalogs, and sidebar.
- tests/skills/test_office_document_skills.py: frontmatter contracts,
  referenced-script existence, schema-map integrity, cross-link
  resolution, script compilation.

E2E validated: docx create->render->edit->validate, xlsx recalc
(SUM + _xlfn.TEXTJOIN evaluate correctly), pdf create->merge->extract,
pptx generate->validate->thumbnail.
2026-07-21 05:39:23 -07:00