Teknium
c49fa88b80
* refactor(skills): shipped-set slim — 15 skills to optional, github six-way merge, pdf absorbs OCR+nano-pdf, channel-gated teams pipeline
Maintainer-directed shipped-skills curation (skills index 1,900 -> ~1,400
tok/call on desktop; every session pays the index, so this is a per-call
diet on all installs):
- optional-skills moves (installable via skills hub, history preserved):
creative comfyui/ascii-art/excalidraw/pretext/sketch/touchdesigner-mcp;
ALL of mlops (huggingface-hub, llama-cpp, serving-llms-vllm,
weights-and-biases, evaluating-llms-harness — subcategory structure
kept); research-paper-writing (55 supporting files, 17.3K-tok load);
openhue; blogwatcher (first taught the cronjob monitor-field watch
pattern + web_extract instead of pre-cron manual workflows)
- DELETED session-librarian (Aug-12 'inspired by Perplexity Computer'
port, never maintainer-intended; session_search covers discovery)
- github: six skills (auth, issues, pr-workflow, issue-to-pr,
code-review, repo-management) merged into ONE software-development/
github skill — routing body + complete per-workflow references;
benbarclay authorship credited; codebase-inspection rides along;
discipline pins from test_github_issue_to_pr_skill.py preserved
against the reference body in the new test_github_skill.py
- pdf absorbs ocr-and-documents + nano-pdf as references/ + scripts
(extract_pymupdf, extract_marker converted to the argparse house
standard its contract test enforces)
- NEW session_platforms frontmatter gate (metadata.hermes): hides a
skill from the index on gateway channels it is not for; fail-open on
unknown platform; teams-meeting-pipeline gated to [teams, cron]
- blocked-page-recovery: research -> new web category; trigger-first
description ('Use when a fetch fails: 403/429, paywall, WAF, bot
wall.') so the model actually reaches for it on blocked fetches
- docs regenerated via generate-skill-docs.py (195 pages); related_skills
swept repo-wide; tests: 1672 passed (2 openclaw failures pre-existing
on clean main, Windows-local)
* chore: ignore .skills_prompt_snapshot.json (local index cache, accidentally committed)
151 lines
2.8 KiB
Markdown
151 lines
2.8 KiB
Markdown
# Server Deployment Guide
|
|
|
|
Production deployment of llama.cpp server with OpenAI-compatible API.
|
|
|
|
## Direct from Hugging Face Hub
|
|
|
|
Prefer the model repo's local-app page first:
|
|
|
|
```text
|
|
https://huggingface.co/<repo>?local-app=llama.cpp
|
|
```
|
|
|
|
If the page shows an exact snippet, copy it. If not, use one of these forms:
|
|
|
|
```bash
|
|
# Choose a quant label directly from the Hub repo
|
|
llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
|
|
```
|
|
|
|
```bash
|
|
# Pin an exact GGUF file from the repo tree
|
|
llama-server \
|
|
--hf-repo microsoft/Phi-3-mini-4k-instruct-gguf \
|
|
--hf-file Phi-3-mini-4k-instruct-q4.gguf \
|
|
-c 4096
|
|
```
|
|
|
|
Use the file-specific form when the repo has custom naming or when you already extracted the exact filename from the tree API.
|
|
|
|
## Server Modes
|
|
|
|
### llama-server
|
|
|
|
```bash
|
|
# Basic server
|
|
./llama-server \
|
|
-m models/llama-2-7b-chat.Q4_K_M.gguf \
|
|
--host 0.0.0.0 \
|
|
--port 8080 \
|
|
-c 4096 # Context size
|
|
|
|
# With GPU acceleration
|
|
./llama-server \
|
|
-m models/llama-2-70b.Q4_K_M.gguf \
|
|
-ngl 40 # Offload 40 layers to GPU
|
|
```
|
|
|
|
## OpenAI-Compatible API
|
|
|
|
### Chat completions
|
|
```bash
|
|
curl http://localhost:8080/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "llama-2",
|
|
"messages": [
|
|
{"role": "system", "content": "You are helpful"},
|
|
{"role": "user", "content": "Hello"}
|
|
],
|
|
"temperature": 0.7,
|
|
"max_tokens": 100
|
|
}'
|
|
```
|
|
|
|
### Streaming
|
|
```bash
|
|
curl http://localhost:8080/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "llama-2",
|
|
"messages": [{"role": "user", "content": "Count to 10"}],
|
|
"stream": true
|
|
}'
|
|
```
|
|
|
|
## Docker Deployment
|
|
|
|
**Dockerfile**:
|
|
```dockerfile
|
|
FROM ubuntu:22.04
|
|
RUN apt-get update && apt-get install -y git build-essential
|
|
RUN git clone https://github.com/ggerganov/llama.cpp
|
|
WORKDIR /llama.cpp
|
|
RUN make LLAMA_CUDA=1
|
|
COPY models/ /models/
|
|
EXPOSE 8080
|
|
CMD ["./llama-server", "-m", "/models/model.gguf", "--host", "0.0.0.0", "--port", "8080"]
|
|
```
|
|
|
|
**Run**:
|
|
```bash
|
|
docker run --gpus all -p 8080:8080 llama-cpp:latest
|
|
```
|
|
|
|
## Monitoring
|
|
|
|
```bash
|
|
# Server metrics endpoint
|
|
curl http://localhost:8080/metrics
|
|
|
|
# Health check
|
|
curl http://localhost:8080/health
|
|
```
|
|
|
|
**Metrics**:
|
|
- requests_total
|
|
- tokens_generated
|
|
- prompt_tokens
|
|
- completion_tokens
|
|
- kv_cache_tokens
|
|
|
|
## Load Balancing
|
|
|
|
**NGINX**:
|
|
```nginx
|
|
upstream llama_cpp {
|
|
server llama1:8080;
|
|
server llama2:8080;
|
|
}
|
|
|
|
server {
|
|
location / {
|
|
proxy_pass http://llama_cpp;
|
|
proxy_read_timeout 300s;
|
|
}
|
|
}
|
|
```
|
|
|
|
## Performance Tuning
|
|
|
|
**Parallel requests**:
|
|
```bash
|
|
./llama-server \
|
|
-m model.gguf \
|
|
-np 4 # 4 parallel slots
|
|
```
|
|
|
|
**Continuous batching**:
|
|
```bash
|
|
./llama-server \
|
|
-m model.gguf \
|
|
--cont-batching # Enable continuous batching
|
|
```
|
|
|
|
**Context caching**:
|
|
```bash
|
|
./llama-server \
|
|
-m model.gguf \
|
|
--cache-prompt # Cache processed prompts
|
|
```
|