Teknium
c49fa88b80
* refactor(skills): shipped-set slim — 15 skills to optional, github six-way merge, pdf absorbs OCR+nano-pdf, channel-gated teams pipeline
Maintainer-directed shipped-skills curation (skills index 1,900 -> ~1,400
tok/call on desktop; every session pays the index, so this is a per-call
diet on all installs):
- optional-skills moves (installable via skills hub, history preserved):
creative comfyui/ascii-art/excalidraw/pretext/sketch/touchdesigner-mcp;
ALL of mlops (huggingface-hub, llama-cpp, serving-llms-vllm,
weights-and-biases, evaluating-llms-harness — subcategory structure
kept); research-paper-writing (55 supporting files, 17.3K-tok load);
openhue; blogwatcher (first taught the cronjob monitor-field watch
pattern + web_extract instead of pre-cron manual workflows)
- DELETED session-librarian (Aug-12 'inspired by Perplexity Computer'
port, never maintainer-intended; session_search covers discovery)
- github: six skills (auth, issues, pr-workflow, issue-to-pr,
code-review, repo-management) merged into ONE software-development/
github skill — routing body + complete per-workflow references;
benbarclay authorship credited; codebase-inspection rides along;
discipline pins from test_github_issue_to_pr_skill.py preserved
against the reference body in the new test_github_skill.py
- pdf absorbs ocr-and-documents + nano-pdf as references/ + scripts
(extract_pymupdf, extract_marker converted to the argparse house
standard its contract test enforces)
- NEW session_platforms frontmatter gate (metadata.hermes): hides a
skill from the index on gateway channels it is not for; fail-open on
unknown platform; teams-meeting-pipeline gated to [teams, cron]
- blocked-page-recovery: research -> new web category; trigger-first
description ('Use when a fetch fails: 403/429, paywall, WAF, bot
wall.') so the model actually reaches for it on blocked fetches
- docs regenerated via generate-skill-docs.py (195 pages); related_skills
swept repo-wide; tests: 1672 passed (2 openclaw failures pre-existing
on clean main, Windows-local)
* chore: ignore .skills_prompt_snapshot.json (local index cache, accidentally committed)
90 lines
1.6 KiB
Markdown
90 lines
1.6 KiB
Markdown
# Performance Optimization Guide
|
||
|
||
Maximize llama.cpp inference speed and efficiency.
|
||
|
||
## CPU Optimization
|
||
|
||
### Thread tuning
|
||
```bash
|
||
# Set threads (default: physical cores)
|
||
./llama-cli -m model.gguf -t 8
|
||
|
||
# For AMD Ryzen 9 7950X (16 cores, 32 threads)
|
||
-t 16 # Best: physical cores
|
||
|
||
# Avoid hyperthreading (slower for matrix ops)
|
||
```
|
||
|
||
### BLAS acceleration
|
||
```bash
|
||
# OpenBLAS (faster matrix ops)
|
||
make LLAMA_OPENBLAS=1
|
||
|
||
# BLAS gives 2-3× speedup
|
||
```
|
||
|
||
## GPU Offloading
|
||
|
||
### Layer offloading
|
||
```bash
|
||
# Offload 35 layers to GPU (hybrid mode)
|
||
./llama-cli -m model.gguf -ngl 35
|
||
|
||
# Offload all layers
|
||
./llama-cli -m model.gguf -ngl 999
|
||
|
||
# Find optimal value:
|
||
# Start with -ngl 999
|
||
# If OOM, reduce by 5 until fits
|
||
```
|
||
|
||
### Memory usage
|
||
```bash
|
||
# Check VRAM usage
|
||
nvidia-smi dmon
|
||
|
||
# Reduce context if needed
|
||
./llama-cli -m model.gguf -c 2048 # 2K context instead of 4K
|
||
```
|
||
|
||
## Batch Processing
|
||
|
||
```bash
|
||
# Increase batch size for throughput
|
||
./llama-cli -m model.gguf -b 512 # Default: 512
|
||
|
||
# Physical batch (GPU)
|
||
--ubatch 128 # Process 128 tokens at once
|
||
```
|
||
|
||
## Context Management
|
||
|
||
```bash
|
||
# Default context (512 tokens)
|
||
-c 512
|
||
|
||
# Longer context (slower, more memory)
|
||
-c 4096
|
||
|
||
# Very long context (if model supports)
|
||
-c 32768
|
||
```
|
||
|
||
## Benchmarks
|
||
|
||
### CPU Performance (Llama 2-7B Q4_K_M)
|
||
|
||
| Setup | Speed | Notes |
|
||
|-------|-------|-------|
|
||
| Apple M3 Max | 50 tok/s | Metal acceleration |
|
||
| AMD 7950X (16c) | 35 tok/s | OpenBLAS |
|
||
| Intel i9-13900K | 30 tok/s | AVX2 |
|
||
|
||
### GPU Offloading (RTX 4090)
|
||
|
||
| Layers GPU | Speed | VRAM |
|
||
|------------|-------|------|
|
||
| 0 (CPU only) | 30 tok/s | 0 GB |
|
||
| 20 (hybrid) | 80 tok/s | 8 GB |
|
||
| 35 (all) | 120 tok/s | 12 GB |
|