Initial commit of EvoScientist framework
Self-evolving AI scientist framework built on LangGraph/LangChain with CLI/TUI core, FastAPI gateway, and Next.js frontend. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
@@ -0,0 +1,27 @@
|
||||
# EvoScientist CLI environment variables
|
||||
# The preferred configuration flow is `evosci onboard`, which writes
|
||||
# ~/.evoscientist/config/settings.yaml. Environment variables can override it.
|
||||
|
||||
# Optional application directories
|
||||
# EVOSCIENTIST_HOME=~/.evoscientist
|
||||
# EVOSCIENTIST_DATA_ROOT=~/.evoscientist/data
|
||||
|
||||
# Logging
|
||||
EVOSCIENTIST_LOG_LEVEL=INFO
|
||||
# EVOSCIENTIST_LOG_DIR=~/.evoscientist/data/logs
|
||||
EVOSCIENTIST_LOG_RETENTION_DAYS=30
|
||||
|
||||
# Optional PostgreSQL checkpoint storage. Without this value the CLI uses an
|
||||
# in-memory checkpointer for the current process.
|
||||
# EVOSCIENTIST_SESSION_DB_URL=postgresql://user:password@localhost:5432/evoscientist
|
||||
|
||||
# Model provider examples. Provider-specific settings can also be configured
|
||||
# through `evosci onboard`.
|
||||
# OPENAI_API_KEY=sk-...
|
||||
# OPENAI_BASE_URL=https://api.openai.com/v1
|
||||
# ANTHROPIC_API_KEY=sk-ant-...
|
||||
# GOOGLE_API_KEY=...
|
||||
# TAVILY_API_KEY=tvly-...
|
||||
|
||||
# Optional default model
|
||||
# DEFAULT_MODEL=openai/gpt-5.4
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: Bug report
|
||||
about: Report a bug to help us improve EvoScientist
|
||||
title: ''
|
||||
labels: bug
|
||||
assignees: ''
|
||||
type: Bug
|
||||
|
||||
---
|
||||
|
||||
**Describe the bug**
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
**To Reproduce**
|
||||
Steps to reproduce the behavior:
|
||||
1. Run `EvoSci ...`
|
||||
2. Provide input '...'
|
||||
3. See error
|
||||
|
||||
**Expected behavior**
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
**Error output / logs**
|
||||
If applicable, paste the relevant terminal output or traceback.
|
||||
|
||||
```
|
||||
<paste here>
|
||||
```
|
||||
|
||||
**System information**
|
||||
- OS: [e.g. Ubuntu 22.04, macOS 14.3, Windows 11]
|
||||
- Python version: [e.g. 3.11.8]
|
||||
- Terminal: [e.g. iTerm2, Alacritty, Windows Terminal]
|
||||
- EvoScientist version: [e.g. 0.0.2 — run `EvoSci --version`]
|
||||
- LLM provider: [e.g. Anthropic, OpenAI, Ollama]
|
||||
|
||||
**Additional context**
|
||||
Add any other context about the problem here.
|
||||
@@ -0,0 +1,8 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: EvoSkills — niche features & custom workflows
|
||||
url: https://github.com/EvoScientist/EvoSkills
|
||||
about: Want to add a feature that serves a specific use case rather than all users? Contribute it as an EvoSkill instead.
|
||||
- name: Community Discord
|
||||
url: https://discord.gg/AZ9ZMXkunY
|
||||
about: Ask questions, get help, and chat with the community.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: Documentation
|
||||
about: Report missing, incorrect, or unclear documentation
|
||||
title: ''
|
||||
labels: documentation
|
||||
assignees: ''
|
||||
type: Documentation
|
||||
|
||||
---
|
||||
|
||||
**What documentation is affected?**
|
||||
Describe which part of the project needs better docs.
|
||||
|
||||
**What is wrong or missing?**
|
||||
Describe the issue — is something inaccurate, outdated, hard to follow, or missing entirely?
|
||||
|
||||
**Suggested improvement**
|
||||
If you have a specific fix or addition in mind, describe it here.
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
name: Feature request
|
||||
about: Suggest an idea for EvoScientist core functionality
|
||||
title: ''
|
||||
labels: enhancement
|
||||
assignees: ''
|
||||
type: Feature
|
||||
|
||||
---
|
||||
|
||||
> **Note:** We only accept feature requests for core functionality that benefits the majority of users.
|
||||
> If your idea is a niche workflow or personal utility, consider contributing it to [EvoSkills](https://github.com/EvoScientist/EvoSkills) instead.
|
||||
|
||||
**Is your feature request related to a problem? Please describe.**
|
||||
A clear and concise description of the problem. Ex. "I'm always frustrated when [...]"
|
||||
|
||||
**Describe the solution you'd like**
|
||||
A clear and concise description of what you want to happen.
|
||||
|
||||
**Describe alternatives you've considered**
|
||||
A clear and concise description of any alternative solutions or features you've considered.
|
||||
|
||||
**Additional context**
|
||||
Add any other context or screenshots about the feature request here.
|
||||
@@ -0,0 +1,21 @@
|
||||
## Description
|
||||
|
||||
<!-- What does this PR do? Link the related issue (e.g. "Closes #123"). -->
|
||||
|
||||
## Type of change
|
||||
|
||||
<!-- Check the one that applies. -->
|
||||
|
||||
- [ ] Bug fix
|
||||
- [ ] New feature — link issue: #<!-- issue number -->
|
||||
- [ ] Documentation / examples
|
||||
- [ ] Test improvement
|
||||
- [ ] Refactor (no behavior change)
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] I have read the [Contributing Guidelines](../CONTRIBUTING.md)
|
||||
- [ ] This targets **core functionality** used by the majority of users (niche features belong in [EvoSkills](https://github.com/EvoScientist/EvoSkills))
|
||||
- [ ] I have added/updated tests where applicable
|
||||
- [ ] `uv run ruff check .` passes
|
||||
- [ ] `uv run pytest` passes
|
||||
|
After Width: | Height: | Size: 849 KiB |
|
After Width: | Height: | Size: 316 KiB |
|
After Width: | Height: | Size: 540 KiB |
|
After Width: | Height: | Size: 284 KiB |
|
After Width: | Height: | Size: 582 KiB |
|
After Width: | Height: | Size: 434 KiB |
|
After Width: | Height: | Size: 642 KiB |
|
After Width: | Height: | Size: 739 KiB |
|
After Width: | Height: | Size: 6.9 MiB |
|
After Width: | Height: | Size: 370 KiB |
|
After Width: | Height: | Size: 492 KiB |
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="188" height="34">
|
||||
<rect x="0.5" y="0.5" width="187" height="33" rx="8" fill="#ffffff" stroke="#e5e7eb"/>
|
||||
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="500" fill="#6b7280">Built with</text>
|
||||
<rect x="84" y="5" width="96" height="24" rx="6" fill="#7c3aed"/>
|
||||
<text x="132" y="22" text-anchor="middle"
|
||||
font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="700" fill="#ffffff">DeepAgents</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 566 B |
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="188" height="34">
|
||||
<rect x="0.5" y="0.5" width="187" height="33" rx="8" fill="#1e293b" stroke="#334155"/>
|
||||
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="500" fill="#94a3b8">Built with</text>
|
||||
<rect x="84" y="5" width="96" height="24" rx="6" fill="#8b5cf6"/>
|
||||
<text x="132" y="22" text-anchor="middle"
|
||||
font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="700" fill="#ffffff">DeepAgents</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 566 B |
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="168" height="34">
|
||||
<rect x="0.5" y="0.5" width="167" height="33" rx="8" fill="#ffffff" stroke="#e5e7eb"/>
|
||||
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="500" fill="#6b7280">License</text>
|
||||
<rect x="71" y="5" width="89" height="24" rx="6" fill="#059669"/>
|
||||
<text x="115.5" y="22" text-anchor="middle"
|
||||
font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="700" fill="#ffffff">Apache 2.0</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 565 B |
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="168" height="34">
|
||||
<rect x="0.5" y="0.5" width="167" height="33" rx="8" fill="#1e293b" stroke="#334155"/>
|
||||
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="500" fill="#94a3b8">License</text>
|
||||
<rect x="71" y="5" width="89" height="24" rx="6" fill="#10b981"/>
|
||||
<text x="115.5" y="22" text-anchor="middle"
|
||||
font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="700" fill="#ffffff">Apache 2.0</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 565 B |
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="124" height="34">
|
||||
<rect x="0.5" y="0.5" width="123" height="33" rx="8" fill="#ffffff" stroke="#e5e7eb"/>
|
||||
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="500" fill="#6b7280">PyPI</text>
|
||||
<rect x="54" y="5" width="62" height="24" rx="6" fill="#1565c0"/>
|
||||
<text x="85" y="22" text-anchor="middle"
|
||||
font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="700" fill="#ffffff">v0.0.7</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 555 B |
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="124" height="34">
|
||||
<rect x="0.5" y="0.5" width="123" height="33" rx="8" fill="#1e293b" stroke="#334155"/>
|
||||
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="500" fill="#94a3b8">PyPI</text>
|
||||
<rect x="54" y="5" width="62" height="24" rx="6" fill="#2563eb"/>
|
||||
<text x="85" y="22" text-anchor="middle"
|
||||
font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="700" fill="#ffffff">v0.0.7</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 555 B |
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="136" height="34">
|
||||
<rect x="0.5" y="0.5" width="135" height="33" rx="8" fill="#ffffff" stroke="#e5e7eb"/>
|
||||
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="500" fill="#6b7280">Website</text>
|
||||
<rect x="74" y="5" width="54" height="24" rx="6" fill="#ea580c"/>
|
||||
<text x="101" y="22" text-anchor="middle"
|
||||
font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="700" fill="#ffffff">Page</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 557 B |
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="136" height="34">
|
||||
<rect x="0.5" y="0.5" width="135" height="33" rx="8" fill="#1e293b" stroke="#334155"/>
|
||||
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="500" fill="#94a3b8">Website</text>
|
||||
<rect x="74" y="5" width="54" height="24" rx="6" fill="#f97316"/>
|
||||
<text x="101" y="22" text-anchor="middle"
|
||||
font-family="Inter, -apple-system, system-ui, sans-serif"
|
||||
font-size="13" font-weight="700" fill="#ffffff">Page</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 557 B |
@@ -0,0 +1,7 @@
|
||||
# 📱 EvoScientist 微信二维码
|
||||
|
||||
### 👥 微信交流群:
|
||||
|
||||
<!--  -->
|
||||
<!-- 3-23 -->
|
||||
<img src="./wechat_group.jpeg" alt="微信群二维码" width="300"/>
|
||||
|
After Width: | Height: | Size: 654 KiB |
|
After Width: | Height: | Size: 94 KiB |
|
After Width: | Height: | Size: 94 KiB |
|
After Width: | Height: | Size: 428 KiB |
|
After Width: | Height: | Size: 13 KiB |
@@ -0,0 +1,24 @@
|
||||
name: Build
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["main"]
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
python-version: "3.11"
|
||||
cache-dependency-glob: "**/pyproject.toml"
|
||||
- name: Build package
|
||||
run: uv build
|
||||
- name: Upload artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: dist
|
||||
path: dist/
|
||||
@@ -0,0 +1,23 @@
|
||||
name: Lint
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["main"]
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
ruff:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
python-version: "3.11"
|
||||
cache-dependency-glob: "**/pyproject.toml"
|
||||
- name: Install dependencies
|
||||
run: uv sync --dev
|
||||
- name: Run ruff linter
|
||||
run: uv run ruff check --output-format=github .
|
||||
- name: Run ruff format
|
||||
run: uv run ruff format --check .
|
||||
@@ -0,0 +1,24 @@
|
||||
name: Test
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["main"]
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
pytest:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ["3.11", "3.12"]
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache-dependency-glob: "**/pyproject.toml"
|
||||
- name: Install dependencies
|
||||
run: uv sync --dev
|
||||
- name: Run pytest
|
||||
run: uv run pytest -v --timeout=30
|
||||
@@ -0,0 +1,67 @@
|
||||
# macOS
|
||||
.DS_Store
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
*.egg
|
||||
*.pytest_cache/
|
||||
.coverage
|
||||
.ipynb_checkpoints/
|
||||
|
||||
# Environment
|
||||
.env
|
||||
.env.*
|
||||
.env_*
|
||||
!.env.example
|
||||
.venv/
|
||||
venv/
|
||||
bridge/node_modules/
|
||||
bridge/package-lock.json
|
||||
|
||||
# IDE / Tools
|
||||
.vscode/
|
||||
.idea/
|
||||
.agents/
|
||||
.claude/
|
||||
.cursor/
|
||||
.codex/
|
||||
.cursorrules
|
||||
.ruff_cache/
|
||||
|
||||
# Project specific
|
||||
.langgraph_api/
|
||||
workspace/
|
||||
skills/
|
||||
memory/
|
||||
media/
|
||||
conversation_history/
|
||||
.deno_cache/
|
||||
*.ipynb
|
||||
*CLAUDE.md
|
||||
*AGENTS.md
|
||||
*meals/
|
||||
botpy.log
|
||||
large_tool_results/
|
||||
|
||||
# Docker runtime data
|
||||
docker/data/
|
||||
|
||||
# Project-level config data
|
||||
.data/
|
||||
|
||||
# Sensitive / credentials (never commit)
|
||||
postgresql:*
|
||||
_s3_backup/
|
||||
|
||||
# Local scratch / tooling data
|
||||
.superpowers/
|
||||
.test-home/
|
||||
tmp/
|
||||
|
||||
# Root-level debug scratch scripts (proper tests live in tests/)
|
||||
/test_*.py
|
||||
/research_lookup_temp.py
|
||||
@@ -0,0 +1,10 @@
|
||||
repos:
|
||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||
# Ruff version.
|
||||
rev: v0.9.9
|
||||
hooks:
|
||||
# Run the linter.
|
||||
- id: ruff
|
||||
args: [ --fix ]
|
||||
# Run the formatter.
|
||||
- id: ruff-format
|
||||
@@ -0,0 +1,111 @@
|
||||
# Contributing to EvoScientist
|
||||
|
||||
We appreciate your interest and the time you spend helping improve EvoScientist. Please read the following guidelines before contributing.
|
||||
|
||||
## How you can contribute
|
||||
|
||||
- **Report bugs and request features:** open an issue using the provided templates. Make sure to use the correct template and labels.
|
||||
- **Propose design changes:** use issues or discussion threads to outline the problem, alternatives, and trade-offs before implementing.
|
||||
- **Contribute code or docs:** submit PRs that address an open issue. They must have a clear rationale and tests where applicable.
|
||||
|
||||
## What we are looking for in PRs
|
||||
|
||||
We aim to keep EvoScientist focused on core functionality that benefits the majority of users. PRs should only include:
|
||||
|
||||
- Bug fixes / improvements to existing features
|
||||
- New features that were proposed in an issue and agreed upon with maintainers
|
||||
- Documentation updates and examples
|
||||
- Meaningful additions to the test suite
|
||||
|
||||
If you want to add a niche or specialized workflow, consider contributing to the [EvoSkills repository](https://github.com/EvoScientist/EvoSkills) instead.
|
||||
|
||||
## Development setup
|
||||
|
||||
1. **Fork and clone** the repository:
|
||||
```bash
|
||||
git clone https://github.com/<your-username>/EvoScientist.git
|
||||
cd EvoScientist
|
||||
```
|
||||
|
||||
2. **Install dependencies** (requires [uv](https://docs.astral.sh/uv/)):
|
||||
```bash
|
||||
uv sync --dev
|
||||
```
|
||||
|
||||
3. **Run the test suite** (no API keys needed):
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
4. **Run the linter:**
|
||||
```bash
|
||||
uv run ruff check .
|
||||
```
|
||||
|
||||
## Submitting a pull request
|
||||
|
||||
1. Create a branch from `main` with a descriptive name (e.g. `fix/session-crash`, `feat/export-csv`).
|
||||
2. Make your changes, keeping commits focused and well-described.
|
||||
3. Ensure `uv run ruff check .` and `uv run pytest` pass locally — these also run in CI.
|
||||
4. Open a PR against `main` and fill in the PR template.
|
||||
5. A maintainer will review your PR. Please be responsive to feedback.
|
||||
|
||||
## Code style
|
||||
|
||||
- We use [Ruff](https://docs.astral.sh/ruff/) for linting. Run `uv run ruff check .` before pushing.
|
||||
- Follow the existing code patterns and conventions in the area you're modifying.
|
||||
- Keep changes minimal and focused on the task at hand.
|
||||
|
||||
---
|
||||
|
||||
## Project overview
|
||||
|
||||
EvoScientist is a multi-agent AI system for automated scientific experimentation and discovery. It orchestrates specialized sub-agents that plan experiments, search literature, write code, debug, analyze data, and draft reports.
|
||||
|
||||
| Fact | Value |
|
||||
|------|-------|
|
||||
| Language | Python 3.11+ |
|
||||
| License | Apache 2.0 |
|
||||
| Framework | [DeepAgents](https://github.com/langchain-ai/deepagents) + [LangChain](https://python.langchain.com/) + [LangGraph](https://langchain-ai.github.io/langgraph/) |
|
||||
| Default model | `claude-sonnet-4-6` (Anthropic) |
|
||||
| Tests | ~890 across 36 files, no API keys needed |
|
||||
| Config file | `.data/.config/settings.yaml` (project root) |
|
||||
|
||||
### Sub-Agents (defined in `EvoScientist/subagent.yaml`)
|
||||
|
||||
| Agent | Purpose |
|
||||
|-------|---------|
|
||||
| `planner-agent` | Creates and updates experimental plans (no web search, no implementation) |
|
||||
| `research-agent` | Web research for methods, baselines, and datasets (Tavily search) |
|
||||
| `code-agent` | Implements experiment code and runnable scripts |
|
||||
| `debug-agent` | Reproduces failures, identifies root causes, applies minimal fixes |
|
||||
| `data-analysis-agent` | Computes metrics, creates plots, summarizes insights |
|
||||
| `writing-agent` | Drafts paper-ready Markdown experiment reports |
|
||||
|
||||
### Data flow
|
||||
|
||||
```txt
|
||||
User Input (CLI / TUI / 10 Channel Integrations)
|
||||
|
|
||||
CLI (cli/) / TUI (cli/tui_*) / Channel Server (channels/)
|
||||
|
|
||||
Main Agent (EvoScientist.py) -- create_deep_agent()
|
||||
+-- System Prompt (prompts.py)
|
||||
+-- Chat Model (llm/ -- multi-provider)
|
||||
+-- Middleware: Memory (middleware/memory.py)
|
||||
+-- Backend: CompositeBackend (backends.py)
|
||||
| / --> CustomSandboxBackend (workspace read/write + execute)
|
||||
| /skills/ --> MergedReadOnlyBackend (user > built-in)
|
||||
| /memory/ --> FilesystemBackend (persistent cross-session)
|
||||
+-- MCP Tools (mcp/ -- optional, cached by config signature)
|
||||
|
|
||||
task tool --> Delegates to Sub-Agents
|
||||
|
|
||||
Stream Events --> Emitter --> Tracker --> State --> Rich Display / TUI
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Need help?
|
||||
|
||||
Reach us on [Discord](https://discord.gg/AZ9ZMXkunY) or WeChat (linked in README).
|
||||
@@ -0,0 +1,766 @@
|
||||
"""EvoScientist Agent graph construction.
|
||||
|
||||
This module defines the agent graph and its factory functions. All heavy
|
||||
initialization (deepagents, backends, LLM, middleware) is deferred to first
|
||||
use so that importing this module is fast and non-agent CLI commands
|
||||
(``EvoSci config list``, ``EvoSci onboard``) never pay the cost.
|
||||
|
||||
Usage:
|
||||
from EvoScientist import EvoScientist_agent
|
||||
|
||||
# Notebook / programmatic usage
|
||||
for state in EvoScientist_agent.stream(
|
||||
{"messages": [HumanMessage(content="your question")]},
|
||||
config={"configurable": {"thread_id": "1"}},
|
||||
stream_mode="values",
|
||||
):
|
||||
...
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from langchain.agents.middleware import AgentMiddleware
|
||||
|
||||
from . import paths as _paths_mod
|
||||
from .config import apply_config_to_env, get_effective_config
|
||||
from .paths import set_active_workspace, set_workspace_root
|
||||
from .path_semantics import path_semantics_prompt
|
||||
from .prompts import (
|
||||
RESEARCHER_INSTRUCTIONS,
|
||||
USER_FACING_LANGUAGE_POLICY,
|
||||
get_system_prompt,
|
||||
)
|
||||
|
||||
# Suppress noisy warnings from deepagents skill loader (non-string frontmatter fields, etc.)
|
||||
logging.getLogger("deepagents.middleware.skills").setLevel(logging.ERROR)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# =============================================================================
|
||||
# Constants
|
||||
# =============================================================================
|
||||
|
||||
SUBAGENTS_CONFIG = Path(__file__).parent / "subagent.yaml"
|
||||
SKILLS_DIR = str(Path(__file__).parent / "skills")
|
||||
|
||||
# =============================================================================
|
||||
# Lazy state — initialized on first use, not at import time
|
||||
# =============================================================================
|
||||
|
||||
_config = None
|
||||
_chat_model = None
|
||||
|
||||
# Cache MCP tools by the effective config signature to avoid reconnecting
|
||||
# to MCP servers on every `/new` when config is unchanged.
|
||||
_MCP_TOOLS_CACHE_KEY: str | None = None
|
||||
_MCP_TOOLS_CACHE_VALUE: dict[str, list] | None = None
|
||||
|
||||
# Default agent (no checkpointer) — used by langgraph dev / LangSmith / notebooks.
|
||||
# Lazily constructed on first access so MCP tools are included without
|
||||
# spawning subprocesses at import time.
|
||||
_EvoScientist_agent = None
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Lazy initialization helpers
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def reset_runtime_caches() -> None:
|
||||
"""Clear cached default runtime objects after configuration changes."""
|
||||
import sys
|
||||
|
||||
global _config, _chat_model, _EvoScientist_agent
|
||||
_config = None
|
||||
_chat_model = None
|
||||
_EvoScientist_agent = None
|
||||
package = sys.modules.get("EvoScientist")
|
||||
if package is not None and "EvoScientist_agent" in vars(package):
|
||||
delattr(package, "EvoScientist_agent")
|
||||
|
||||
|
||||
def _ensure_config(config=None):
|
||||
"""Return cached config. If *config* is passed, cache and use it."""
|
||||
global _config
|
||||
if config is not None:
|
||||
_config = config
|
||||
apply_config_to_env(_config)
|
||||
if _config is None:
|
||||
_config = get_effective_config()
|
||||
apply_config_to_env(_config)
|
||||
return _config
|
||||
|
||||
|
||||
def _ensure_chat_model():
|
||||
"""Return cached chat model, creating it on first call."""
|
||||
global _chat_model
|
||||
if _chat_model is None:
|
||||
from .llm import get_chat_model
|
||||
|
||||
cfg = _ensure_config()
|
||||
_chat_model = get_chat_model(model=cfg.model, provider=cfg.provider)
|
||||
return _chat_model
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# MCP caching
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def _load_mcp_config_once() -> tuple[str, dict]:
|
||||
"""Load MCP config and return ``(signature, config)``.
|
||||
|
||||
The signature includes the active workspace directory so that MCP
|
||||
subprocesses are spawned with the correct ``cwd`` and environment
|
||||
for each isolated workspace (e.g. per web thread).
|
||||
"""
|
||||
from .mcp.client import load_mcp_config
|
||||
from .paths import _active_workspace
|
||||
|
||||
cfg = load_mcp_config()
|
||||
if not cfg:
|
||||
logger.info("MCP startup: no MCP servers configured")
|
||||
return "", {}
|
||||
logger.info(
|
||||
"MCP startup: loaded %d server config(s): %s",
|
||||
len(cfg),
|
||||
", ".join(cfg.keys()),
|
||||
)
|
||||
try:
|
||||
sig = json.dumps({"mcp": cfg, "workspace": str(_active_workspace)}, sort_keys=True, ensure_ascii=True)
|
||||
except TypeError:
|
||||
sig = repr(cfg) + str(_active_workspace)
|
||||
return sig, cfg
|
||||
|
||||
|
||||
def _load_mcp_tools_cached() -> dict[str, list]:
|
||||
"""Load MCP tools with config + workspace-aware caching.
|
||||
|
||||
MCP subprocesses are spawned with ``cwd`` and ``EVOSCIENTIST_WORKSPACE_DIR``
|
||||
set to the active workspace. Including the workspace in the cache key
|
||||
ensures that each isolated workspace (e.g. per web thread) gets its own
|
||||
MCP subprocess with the correct working directory.
|
||||
"""
|
||||
global _MCP_TOOLS_CACHE_KEY, _MCP_TOOLS_CACHE_VALUE
|
||||
|
||||
from .mcp import load_mcp_tools
|
||||
|
||||
cfg_key, cfg = _load_mcp_config_once()
|
||||
if not cfg_key:
|
||||
_MCP_TOOLS_CACHE_KEY = ""
|
||||
_MCP_TOOLS_CACHE_VALUE = {}
|
||||
return {}
|
||||
|
||||
if _MCP_TOOLS_CACHE_KEY == cfg_key and _MCP_TOOLS_CACHE_VALUE is not None:
|
||||
logger.info(
|
||||
"MCP startup: using cached MCP tools for %d agent route(s)",
|
||||
len(_MCP_TOOLS_CACHE_VALUE),
|
||||
)
|
||||
return {k: list(v) for k, v in _MCP_TOOLS_CACHE_VALUE.items()}
|
||||
|
||||
loaded = load_mcp_tools(config=cfg)
|
||||
if not loaded:
|
||||
logger.warning("MCP startup: no tools loaded from configured MCP servers")
|
||||
_MCP_TOOLS_CACHE_KEY = cfg_key
|
||||
_MCP_TOOLS_CACHE_VALUE = {}
|
||||
return {}
|
||||
_MCP_TOOLS_CACHE_KEY = cfg_key
|
||||
_MCP_TOOLS_CACHE_VALUE = {k: list(v) for k, v in loaded.items()}
|
||||
return {k: list(v) for k, v in loaded.items()}
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Agent construction helpers
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def _inject_subagent_middleware(subs: list[dict]) -> None:
|
||||
"""Ensure every subagent gets error handling and context management middleware.
|
||||
|
||||
Without this, subagent tool errors are caught by LangGraph's default
|
||||
ToolNode handler which produces terse messages without tracebacks or
|
||||
retry guidance — reducing the subagent's ability to self-recover.
|
||||
"""
|
||||
from .middleware import (
|
||||
ContextOverflowMapperMiddleware,
|
||||
ToolErrorHandlerMiddleware,
|
||||
create_context_editing_middleware,
|
||||
)
|
||||
|
||||
for sa in subs:
|
||||
sa.setdefault("middleware", []).extend(
|
||||
[
|
||||
# Uses main agent's model for trigger — subagents currently
|
||||
# share the same model, so context window matches.
|
||||
create_context_editing_middleware(),
|
||||
ToolErrorHandlerMiddleware(),
|
||||
ContextOverflowMapperMiddleware(),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _inject_subagent_language_policy(subs: list[dict]) -> None:
|
||||
"""Append the shared user-facing language policy to every subagent."""
|
||||
for sa in subs:
|
||||
prompt = str(sa.get("system_prompt") or "")
|
||||
if "## User-Facing Language Policy" not in prompt:
|
||||
sa["system_prompt"] = prompt.rstrip() + "\n\n" + USER_FACING_LANGUAGE_POLICY
|
||||
|
||||
|
||||
def _inject_subagent_path_policy(subs: list[dict]) -> None:
|
||||
"""Append the shared path semantics policy to every subagent."""
|
||||
policy = path_semantics_prompt()
|
||||
for sa in subs:
|
||||
prompt = str(sa.get("system_prompt") or "")
|
||||
if "## Path Semantics" not in prompt:
|
||||
sa["system_prompt"] = prompt.rstrip() + "\n\n" + policy
|
||||
|
||||
|
||||
def _build_prompt_refs() -> dict:
|
||||
"""Build prompt references with the current date (not frozen at import)."""
|
||||
return {
|
||||
"RESEARCHER_INSTRUCTIONS": RESEARCHER_INSTRUCTIONS.format(
|
||||
date=datetime.now().strftime("%Y-%m-%d"),
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _build_base_kwargs(base_backend, base_middleware, *, chat_model=None):
|
||||
"""Build agent kwargs *without* MCP (fast, no subprocess spawning)."""
|
||||
from .tools import (
|
||||
edit_image,
|
||||
generate_image,
|
||||
refresh_image_tool_descriptions,
|
||||
skill_manager,
|
||||
think_tool,
|
||||
web_crawl,
|
||||
web_extract,
|
||||
web_search,
|
||||
)
|
||||
from .utils import load_subagents
|
||||
|
||||
tool_registry = {"think_tool": think_tool}
|
||||
# Register unified search tools when any search backend key is available
|
||||
_has_search = any(os.environ.get(k) for k in (
|
||||
"ZHIPU_API_KEY", "TAVILY_API_KEY", "FIRECRAWL_API_KEY", "FIRECRAWL_API_URL", "EXA_API_KEY",
|
||||
))
|
||||
if _has_search:
|
||||
tool_registry["web_search"] = web_search
|
||||
tool_registry["web_extract"] = web_extract
|
||||
tool_registry["web_crawl"] = web_crawl
|
||||
refresh_image_tool_descriptions()
|
||||
base_tools = [think_tool, skill_manager, generate_image, edit_image]
|
||||
|
||||
subs = load_subagents(
|
||||
SUBAGENTS_CONFIG,
|
||||
tool_registry=tool_registry,
|
||||
prompt_refs=_build_prompt_refs(),
|
||||
)
|
||||
_inject_subagent_path_policy(subs)
|
||||
_inject_subagent_language_policy(subs)
|
||||
_inject_subagent_middleware(subs)
|
||||
return {
|
||||
"name": "EvoScientist",
|
||||
"model": chat_model if chat_model is not None else _ensure_chat_model(),
|
||||
"tools": list(base_tools),
|
||||
"backend": base_backend,
|
||||
"subagents": subs,
|
||||
"middleware": base_middleware,
|
||||
"system_prompt": get_system_prompt(has_parse_documents=False),
|
||||
"skills": ["/skills/"],
|
||||
}
|
||||
|
||||
|
||||
def _apply_conversation_profile(kwargs: dict, conversation_mode: str | None) -> dict:
|
||||
"""Apply per-request capability limits without mutating shared inputs."""
|
||||
from .conversation_modes import get_conversation_profile
|
||||
|
||||
profile = get_conversation_profile(conversation_mode)
|
||||
if profile.enable_subagents:
|
||||
return kwargs
|
||||
adjusted = dict(kwargs)
|
||||
adjusted["subagents"] = []
|
||||
return adjusted
|
||||
|
||||
|
||||
def _resolve_reasoning_effort(effort: str | None, cfg) -> str:
|
||||
"""Resolve reasoning effort: explicit value wins, else fall back to config.
|
||||
|
||||
Decoupled from conversation mode — the profile no longer carries a
|
||||
reasoning_effort default, so the fallback is the top-level config field.
|
||||
"""
|
||||
if effort is not None:
|
||||
return effort
|
||||
return getattr(cfg, "reasoning_effort", "medium")
|
||||
|
||||
|
||||
def _create_tool_selector_for_profile(factory, threshold: int):
|
||||
"""Call tool-selector factory with threshold, tolerating legacy test doubles."""
|
||||
try:
|
||||
return factory(threshold=threshold)
|
||||
except TypeError as exc:
|
||||
if "threshold" not in str(exc) and "unexpected keyword" not in str(exc):
|
||||
raise
|
||||
return factory()
|
||||
|
||||
|
||||
def load_mcp_and_build_kwargs(base_backend, base_middleware, *, chat_model=None):
|
||||
"""Load MCP tools (cached by config) and build agent kwargs.
|
||||
|
||||
Re-connects to MCP servers only when the effective MCP config changes.
|
||||
Falls back to base kwargs if no MCP configured.
|
||||
"""
|
||||
from .tools import (
|
||||
edit_image,
|
||||
generate_image,
|
||||
refresh_image_tool_descriptions,
|
||||
skill_manager,
|
||||
think_tool,
|
||||
web_crawl,
|
||||
web_extract,
|
||||
web_search,
|
||||
)
|
||||
from .utils import load_subagents
|
||||
|
||||
mcp_by_agent = _load_mcp_tools_cached()
|
||||
if not mcp_by_agent:
|
||||
logger.info("MCP startup: agent will start without MCP tools")
|
||||
return _build_base_kwargs(base_backend, base_middleware, chat_model=chat_model)
|
||||
|
||||
tool_registry = {"think_tool": think_tool}
|
||||
# Register unified search tools when any search backend key is available
|
||||
_has_search = any(os.environ.get(k) for k in (
|
||||
"ZHIPU_API_KEY", "TAVILY_API_KEY", "FIRECRAWL_API_KEY", "FIRECRAWL_API_URL", "EXA_API_KEY",
|
||||
))
|
||||
if _has_search:
|
||||
tool_registry["web_search"] = web_search
|
||||
tool_registry["web_extract"] = web_extract
|
||||
tool_registry["web_crawl"] = web_crawl
|
||||
refresh_image_tool_descriptions()
|
||||
base_tools = [think_tool, skill_manager, generate_image, edit_image]
|
||||
|
||||
# Fresh tool registry — start from base tools + MCP tools
|
||||
registry = dict(tool_registry)
|
||||
for tools in mcp_by_agent.values():
|
||||
for t in tools:
|
||||
registry[t.name] = t
|
||||
|
||||
mcp_main = mcp_by_agent.pop("main", [])
|
||||
has_parse_documents = any(t.name == "parse_documents" for t in mcp_main)
|
||||
logger.info(
|
||||
"MCP startup: injecting %d MCP tool(s) into main agent: %s",
|
||||
len(mcp_main),
|
||||
", ".join(t.name for t in mcp_main) or "(none)",
|
||||
)
|
||||
|
||||
subs = load_subagents(
|
||||
SUBAGENTS_CONFIG,
|
||||
tool_registry=registry,
|
||||
prompt_refs=_build_prompt_refs(),
|
||||
)
|
||||
|
||||
_inject_subagent_language_policy(subs)
|
||||
_inject_subagent_path_policy(subs)
|
||||
_inject_subagent_middleware(subs)
|
||||
|
||||
# Inject MCP tools into subagents by name
|
||||
for sa in subs:
|
||||
if sa_tools := mcp_by_agent.get(sa["name"], []):
|
||||
sa.setdefault("tools", []).extend(sa_tools)
|
||||
logger.info(
|
||||
"MCP startup: injecting %d MCP tool(s) into subagent %s: %s",
|
||||
len(sa_tools),
|
||||
sa["name"],
|
||||
", ".join(t.name for t in sa_tools),
|
||||
)
|
||||
|
||||
return {
|
||||
"name": "EvoScientist",
|
||||
"model": chat_model if chat_model is not None else _ensure_chat_model(),
|
||||
"tools": base_tools + mcp_main,
|
||||
"backend": base_backend,
|
||||
"subagents": subs,
|
||||
"middleware": base_middleware,
|
||||
"system_prompt": get_system_prompt(has_parse_documents=has_parse_documents),
|
||||
"skills": ["/skills/"],
|
||||
}
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Default agent (langgraph dev / notebooks)
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def _get_default_backend():
|
||||
"""Build the default composite backend from current paths."""
|
||||
from deepagents.backends import CompositeBackend, FilesystemBackend
|
||||
|
||||
from .backends import CustomSandboxBackend, MergedReadOnlyBackend
|
||||
|
||||
workspace_dir = str(_paths_mod.WORKSPACE_ROOT)
|
||||
set_active_workspace(workspace_dir)
|
||||
memory_dir = str(_paths_mod.MEMORY_DIR)
|
||||
user_skills_dir = str(_paths_mod.USER_SKILLS_DIR)
|
||||
global_skills_dir = str(_paths_mod.GLOBAL_SKILLS_DIR)
|
||||
|
||||
ws_backend = CustomSandboxBackend(
|
||||
root_dir=workspace_dir,
|
||||
virtual_mode=True,
|
||||
timeout=300,
|
||||
)
|
||||
sk_backend = MergedReadOnlyBackend(
|
||||
primary_dir=user_skills_dir,
|
||||
global_dir=global_skills_dir,
|
||||
secondary_dir=SKILLS_DIR,
|
||||
)
|
||||
mem_backend = FilesystemBackend(
|
||||
root_dir=memory_dir,
|
||||
virtual_mode=True,
|
||||
)
|
||||
return CompositeBackend(
|
||||
default=ws_backend,
|
||||
routes={
|
||||
"/skills/": sk_backend,
|
||||
"/memory/": mem_backend,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _get_default_middleware():
|
||||
"""Build the default middleware list."""
|
||||
from .middleware import (
|
||||
ContextOverflowMapperMiddleware,
|
||||
ToolErrorHandlerMiddleware,
|
||||
create_context_editing_middleware,
|
||||
create_memory_middleware,
|
||||
create_tool_selector_middleware,
|
||||
)
|
||||
|
||||
cfg = _ensure_config()
|
||||
model = _ensure_chat_model()
|
||||
memory_dir = str(_paths_mod.MEMORY_DIR)
|
||||
mw: list[AgentMiddleware] = [
|
||||
create_context_editing_middleware(model),
|
||||
ContextOverflowMapperMiddleware(),
|
||||
ToolErrorHandlerMiddleware(),
|
||||
*create_tool_selector_middleware(),
|
||||
create_memory_middleware(memory_dir, extraction_model=model),
|
||||
]
|
||||
|
||||
if cfg.enable_ask_user and not cfg.auto_mode:
|
||||
from .middleware.ask_user import AskUserMiddleware
|
||||
|
||||
mw.insert(0, AskUserMiddleware())
|
||||
return mw
|
||||
|
||||
|
||||
def _get_default_agent():
|
||||
"""Build the default agent (with MCP, no checkpointer) on first access."""
|
||||
global _EvoScientist_agent
|
||||
if _EvoScientist_agent is None:
|
||||
from deepagents import create_deep_agent
|
||||
|
||||
be = _get_default_backend()
|
||||
mw = _get_default_middleware()
|
||||
kwargs = load_mcp_and_build_kwargs(be, mw)
|
||||
_EvoScientist_agent = create_deep_agent(**kwargs).with_config(
|
||||
{"recursion_limit": 1000}
|
||||
)
|
||||
return _EvoScientist_agent
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
if name == "EvoScientist_agent":
|
||||
return _get_default_agent()
|
||||
# Backward compat for module-level names
|
||||
if name == "chat_model":
|
||||
return _ensure_chat_model()
|
||||
if name == "SYSTEM_PROMPT":
|
||||
return get_system_prompt(has_parse_documents=False)
|
||||
if name == "backend":
|
||||
return _get_default_backend()
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# CLI agent factory
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def create_cli_agent(
|
||||
workspace_dir: str | None = None,
|
||||
checkpointer=None,
|
||||
config=None,
|
||||
memory_dir: str | None = None,
|
||||
source: str = "cli",
|
||||
user_id: str | None = None,
|
||||
model: str | None = None,
|
||||
provider: str | None = None,
|
||||
reasoning_effort: str | None = None,
|
||||
# ── Compute (Phase 2B v1.6) ─────────────────────────────────────
|
||||
compute_client=None, # ComputeClient for remote execution
|
||||
storage_service=None, # StorageService for object storage
|
||||
app_state=None, # Gateway FastAPI app.state
|
||||
plan: str = "starter", # User plan for quota checks
|
||||
thread_id: str = "", # Thread ID for compute billing
|
||||
compute_quota_ok: bool = False,
|
||||
remaining_compute_minutes: int = 0,
|
||||
input_objects: list | None = None,
|
||||
artifact_prefix: str = "",
|
||||
default_output_paths: list[str] | None = None,
|
||||
max_runtime_seconds: int | None = None,
|
||||
budget_snapshot: dict | None = None,
|
||||
storage_backend: str = "nfs",
|
||||
execution_mode: str = "standard",
|
||||
resource_class: str = "small",
|
||||
backend_policy: str = "auto",
|
||||
compute_fallback_mode: str = "local_sandbox",
|
||||
conversation_mode: str | None = None,
|
||||
_backend_ref: dict | None = None,
|
||||
):
|
||||
"""Create agent with checkpointer for CLI multi-turn support.
|
||||
|
||||
A fresh backend is constructed on every call using the current
|
||||
``paths.WORKSPACE_ROOT`` (or the explicit *workspace_dir*), so
|
||||
runtime ``set_workspace_root()`` changes are always respected.
|
||||
|
||||
Args:
|
||||
workspace_dir: Per-session workspace directory. If ``None``,
|
||||
defaults to the current ``paths.WORKSPACE_ROOT``.
|
||||
checkpointer: Optional LangGraph checkpointer. If ``None``,
|
||||
falls back to ``InMemorySaver`` (non-persistent).
|
||||
config: Optional pre-loaded ``EvoScientistConfig``. If ``None``,
|
||||
loads from file/env/defaults. Passing this avoids double
|
||||
loading when the CLI has already loaded config.
|
||||
memory_dir: Optional memory directory override. If ``None``,
|
||||
uses the default global memory path. Gateway passes per-user path.
|
||||
source: Thread source identifier (\"cli\" or \"web\"). Default \"cli\".
|
||||
user_id: Optional user identifier for multi-tenant isolation.
|
||||
model: Optional model override (short name or full ID). Gateway
|
||||
passes the user-selected model from the frontend.
|
||||
provider: Optional provider override. Inferred from *model* if
|
||||
not specified.
|
||||
reasoning_effort: Optional reasoning effort level for OpenAI models.
|
||||
One of "low", "medium", "high". Passed to the LLM as
|
||||
``reasoning={"effort": value, "summary": "auto"}``.
|
||||
"""
|
||||
import os as _os
|
||||
|
||||
from deepagents import create_deep_agent
|
||||
from deepagents.backends import CompositeBackend, FilesystemBackend
|
||||
|
||||
from . import paths as _paths
|
||||
from .backends import (
|
||||
CustomSandboxBackend,
|
||||
MergedReadOnlyBackend,
|
||||
MultiRootSandboxBackend,
|
||||
NonComputingFallbackBackend,
|
||||
)
|
||||
from .middleware import (
|
||||
ContextOverflowMapperMiddleware,
|
||||
ToolErrorHandlerMiddleware,
|
||||
create_context_editing_middleware,
|
||||
create_memory_middleware,
|
||||
create_tool_selector_middleware,
|
||||
ToolResultLimiterMiddleware,
|
||||
)
|
||||
|
||||
cfg = _ensure_config(config)
|
||||
from .conversation_modes import (
|
||||
get_conversation_profile,
|
||||
reset_current_conversation_mode,
|
||||
set_current_conversation_mode,
|
||||
)
|
||||
|
||||
mode_profile = get_conversation_profile(conversation_mode or getattr(cfg, "conversation_mode", "medium"))
|
||||
|
||||
if checkpointer is None:
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
checkpointer = InMemorySaver()
|
||||
|
||||
# When no explicit workspace_dir is provided, apply config.default_workdir
|
||||
# as a fallback. This covers direct callers (notebooks, iMessage server)
|
||||
# that never call set_workspace_root() themselves. CLI callers always
|
||||
# pass workspace_dir explicitly, so their --workdir is never overwritten.
|
||||
if workspace_dir is None:
|
||||
if cfg.default_workdir:
|
||||
set_workspace_root(
|
||||
_os.path.abspath(_os.path.expanduser(cfg.default_workdir))
|
||||
)
|
||||
workspace_dir = str(_paths.WORKSPACE_ROOT)
|
||||
|
||||
# Read paths dynamically so runtime set_workspace_root() changes are picked up
|
||||
_mem_dir = memory_dir or str(_paths.MEMORY_DIR)
|
||||
_usr_skills_dir = str(_paths.USER_SKILLS_DIR)
|
||||
_global_skills_dir = str(_paths.GLOBAL_SKILLS_DIR)
|
||||
|
||||
# Always construct fresh backends from current paths (avoids stale
|
||||
# module-level backend when workspace root changed at runtime).
|
||||
set_active_workspace(workspace_dir)
|
||||
|
||||
if source == "web" and user_id:
|
||||
# Web user: use remote compute only when the v1.6 preflight approved it.
|
||||
if (
|
||||
compute_client is not None
|
||||
and getattr(compute_client, "ready", False)
|
||||
and compute_quota_ok
|
||||
and storage_backend in ("nfs", "file_gateway")
|
||||
):
|
||||
from .container_sandbox import ContainerSandboxBackend
|
||||
|
||||
ws_backend = ContainerSandboxBackend(
|
||||
compute_client=compute_client,
|
||||
user_id=str(user_id),
|
||||
thread_id=thread_id,
|
||||
input_objects=input_objects or [],
|
||||
artifact_prefix=artifact_prefix,
|
||||
default_output_paths=default_output_paths or ["/workspace/artifacts"],
|
||||
max_runtime_seconds=max_runtime_seconds,
|
||||
budget_snapshot=budget_snapshot or {},
|
||||
execution_mode=execution_mode,
|
||||
resource_class=resource_class,
|
||||
backend_policy=backend_policy,
|
||||
)
|
||||
if _backend_ref is not None:
|
||||
_backend_ref["backend"] = ws_backend
|
||||
else:
|
||||
# Fallback: file browsing/recovery only. Web shell execution is disabled.
|
||||
from .paths import global_data_dir, user_data_dir
|
||||
|
||||
_uid = str(user_id)
|
||||
if storage_backend == "file_gateway":
|
||||
try:
|
||||
from .runtime_integrations import resolve_user_storage_root
|
||||
|
||||
_user_root = resolve_user_storage_root(_uid)
|
||||
if _user_root is None:
|
||||
raise RuntimeError("No external user storage provider is configured")
|
||||
_user_global = _user_root / "__global__"
|
||||
except Exception:
|
||||
# The mounted thread workspace is authoritative in
|
||||
# FileGateway mode, even when the host integration provider
|
||||
# is unavailable (for example in isolated tests/tools).
|
||||
_user_root = Path(workspace_dir).resolve().parent
|
||||
_user_global = _user_root / "__global__"
|
||||
else:
|
||||
_user_root = user_data_dir(_uid)
|
||||
_user_global = global_data_dir(_uid)
|
||||
_thread_root = Path(workspace_dir).resolve()
|
||||
|
||||
# Collect peer thread dirs (same user, other threads) as read-only sources
|
||||
_peer_threads: list[str] = []
|
||||
if _user_root.is_dir():
|
||||
for d in sorted(_user_root.iterdir()):
|
||||
if d.is_dir() and d.name != "__global__" and d.resolve() != _thread_root:
|
||||
_peer_threads.append(str(d))
|
||||
|
||||
reason = "remote compute is unavailable"
|
||||
if storage_backend not in ("nfs", "file_gateway"):
|
||||
reason = f"storage backend {storage_backend!r} is unsupported for Web compute"
|
||||
elif not compute_quota_ok:
|
||||
reason = "compute quota preflight failed"
|
||||
fallback_mode = (
|
||||
compute_fallback_mode
|
||||
or os.getenv("COMPUTE_FALLBACK_MODE", "local_sandbox")
|
||||
or "local_sandbox"
|
||||
).strip().lower()
|
||||
if fallback_mode == "local_sandbox":
|
||||
logger.warning(
|
||||
"web_remote_compute_unavailable_using_local_sandbox thread=%s reason=%s",
|
||||
thread_id,
|
||||
reason,
|
||||
)
|
||||
ws_backend = MultiRootSandboxBackend(
|
||||
write_root=str(_thread_root),
|
||||
read_roots=_peer_threads,
|
||||
global_root=str(_user_global),
|
||||
virtual_mode=True,
|
||||
timeout=max_runtime_seconds or 300,
|
||||
)
|
||||
else:
|
||||
ws_backend = NonComputingFallbackBackend(
|
||||
write_root=str(_thread_root),
|
||||
read_roots=_peer_threads,
|
||||
global_root=str(_user_global),
|
||||
virtual_mode=True,
|
||||
reason=reason,
|
||||
)
|
||||
else:
|
||||
# CLI user: isolated sandbox (unchanged behavior)
|
||||
ws_backend = CustomSandboxBackend(
|
||||
root_dir=workspace_dir,
|
||||
virtual_mode=True,
|
||||
timeout=300,
|
||||
)
|
||||
sk_backend = MergedReadOnlyBackend(
|
||||
primary_dir=_usr_skills_dir,
|
||||
global_dir=_global_skills_dir,
|
||||
secondary_dir=SKILLS_DIR,
|
||||
)
|
||||
# Memory always uses SHARED directory (not per-session) for cross-session persistence
|
||||
mem_backend = FilesystemBackend(
|
||||
root_dir=_mem_dir,
|
||||
virtual_mode=True,
|
||||
)
|
||||
be = CompositeBackend(
|
||||
default=ws_backend,
|
||||
routes={
|
||||
"/skills/": sk_backend,
|
||||
"/memory/": mem_backend,
|
||||
},
|
||||
)
|
||||
|
||||
# Create chat model — use override if provided, otherwise use cached default
|
||||
reasoning_effort = _resolve_reasoning_effort(reasoning_effort, cfg)
|
||||
|
||||
if model:
|
||||
from .llm import get_chat_model
|
||||
|
||||
chat_model = get_chat_model(model=model, provider=provider)
|
||||
else:
|
||||
chat_model = _ensure_chat_model()
|
||||
|
||||
mw: list[AgentMiddleware] = [
|
||||
create_context_editing_middleware(chat_model),
|
||||
ContextOverflowMapperMiddleware(),
|
||||
ToolErrorHandlerMiddleware(),
|
||||
ToolResultLimiterMiddleware(mode_profile.max_tool_result_chars),
|
||||
*_create_tool_selector_for_profile(
|
||||
create_tool_selector_middleware,
|
||||
mode_profile.tool_selector_threshold,
|
||||
),
|
||||
create_memory_middleware(
|
||||
_mem_dir,
|
||||
extraction_model=chat_model,
|
||||
inject_memory=mode_profile.memory_injection_enabled,
|
||||
max_injected_chars=mode_profile.memory_injection_max_chars,
|
||||
),
|
||||
]
|
||||
if cfg.enable_ask_user and not cfg.auto_mode:
|
||||
from .middleware.ask_user import AskUserMiddleware
|
||||
|
||||
mw.insert(0, AskUserMiddleware())
|
||||
|
||||
# Re-load MCP tools from current config (picks up /mcp add changes)
|
||||
kwargs = load_mcp_and_build_kwargs(be, mw, chat_model=chat_model)
|
||||
kwargs = _apply_conversation_profile(kwargs, mode_profile.mode)
|
||||
|
||||
# HITL: gate shell execution for user approval
|
||||
_interrupt_on: dict[str, bool] | None = None
|
||||
if not cfg.auto_approve:
|
||||
_interrupt_on = {"execute": True}
|
||||
|
||||
mode_token = set_current_conversation_mode(mode_profile.mode)
|
||||
try:
|
||||
return create_deep_agent(
|
||||
**kwargs,
|
||||
checkpointer=checkpointer,
|
||||
interrupt_on=_interrupt_on,
|
||||
).with_config({"recursion_limit": 1000})
|
||||
finally:
|
||||
reset_current_conversation_mode(mode_token)
|
||||
@@ -0,0 +1,79 @@
|
||||
"""EvoScientist Agent - AI-powered research and code execution.
|
||||
|
||||
This package exposes a convenience API at the package root while keeping
|
||||
imports lazy, so lightweight modules (for example config helpers) can be used
|
||||
without importing heavy runtime dependencies.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from importlib import import_module
|
||||
|
||||
from ._version import __version__
|
||||
|
||||
_EXPORTS: dict[str, tuple[str, str]] = {
|
||||
# Agent graph (lazy to avoid expensive initialization at import time)
|
||||
"EvoScientist_agent": (".EvoScientist", "EvoScientist_agent"),
|
||||
"create_cli_agent": (".EvoScientist", "create_cli_agent"),
|
||||
# Backends
|
||||
"CustomSandboxBackend": (".backends", "CustomSandboxBackend"),
|
||||
"ReadOnlyFilesystemBackend": (".backends", "ReadOnlyFilesystemBackend"),
|
||||
# Configuration
|
||||
"EvoScientistConfig": (".config", "EvoScientistConfig"),
|
||||
"load_config": (".config", "load_config"),
|
||||
"save_config": (".config", "save_config"),
|
||||
"get_effective_config": (".config", "get_effective_config"),
|
||||
"get_config_path": (".config", "get_config_path"),
|
||||
# LLM
|
||||
"get_chat_model": (".llm", "get_chat_model"),
|
||||
"list_models": (".llm", "list_models"),
|
||||
# Prompts
|
||||
"get_system_prompt": (".prompts", "get_system_prompt"),
|
||||
"RESEARCHER_INSTRUCTIONS": (".prompts", "RESEARCHER_INSTRUCTIONS"),
|
||||
# Tools
|
||||
"tavily_search": (".tools", "tavily_search"),
|
||||
"web_search": (".tools", "web_search"),
|
||||
"web_extract": (".tools", "web_extract"),
|
||||
"web_crawl": (".tools", "web_crawl"),
|
||||
"think_tool": (".tools", "think_tool"),
|
||||
# Sessions
|
||||
"get_checkpointer": (".sessions", "get_checkpointer"),
|
||||
"generate_thread_id": (".sessions", "generate_thread_id"),
|
||||
"list_threads": (".sessions", "list_threads"),
|
||||
"delete_thread": (".sessions", "delete_thread"),
|
||||
"get_storage_stats": (".sessions", "get_storage_stats"),
|
||||
"get_aggregated_storage_stats": (".sessions", "get_aggregated_storage_stats"),
|
||||
"list_all_session_db_paths": (".sessions", "list_all_session_db_paths"),
|
||||
}
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
"""Lazily import and cache package-level attributes.
|
||||
|
||||
Args:
|
||||
name: The attribute name to look up.
|
||||
|
||||
Returns:
|
||||
The resolved attribute value.
|
||||
|
||||
Raises:
|
||||
AttributeError: If the name is not in _EXPORTS.
|
||||
"""
|
||||
target = _EXPORTS.get(name)
|
||||
if target is None:
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
|
||||
module_name, attr_name = target
|
||||
module = import_module(module_name, package=__name__)
|
||||
value = getattr(module, attr_name)
|
||||
# Cache after first load to avoid repeated import lookups.
|
||||
globals()[name] = value
|
||||
return value
|
||||
|
||||
|
||||
def __dir__() -> list[str]:
|
||||
"""List available public attributes including lazy exports."""
|
||||
return sorted(set(globals()) | set(_EXPORTS))
|
||||
|
||||
|
||||
__all__ = list(_EXPORTS)
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Enable `python -m EvoScientist` execution."""
|
||||
|
||||
from EvoScientist.cli import main
|
||||
|
||||
main()
|
||||
@@ -0,0 +1 @@
|
||||
__version__ = "0.1.19"
|
||||
@@ -0,0 +1,348 @@
|
||||
"""ccproxy lifecycle management for OAuth-based Anthropic access.
|
||||
|
||||
Provides functions to start/stop/health-check ccproxy, which allows
|
||||
users with a Claude Pro/Max subscription to use EvoScientist without
|
||||
a separate API key by reusing Claude Code's OAuth tokens.
|
||||
|
||||
ccproxy is invoked via subprocess (not Python imports) so the
|
||||
``ccproxy-api`` package is truly optional at runtime.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
from EvoScientist.config import EvoScientistConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Availability & auth checks
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def _ccproxy_exe() -> str | None:
|
||||
"""Return the path to the ccproxy binary, or None if not found.
|
||||
|
||||
Checks PATH first, then the current Python environment's bin directory
|
||||
(handles conda envs where newly installed binaries may not be visible
|
||||
to shutil.which immediately after pip install).
|
||||
"""
|
||||
found = shutil.which("ccproxy")
|
||||
if found:
|
||||
return found
|
||||
import sys as _sys
|
||||
|
||||
candidate = os.path.join(os.path.dirname(_sys.executable), "ccproxy")
|
||||
if os.path.isfile(candidate) and os.access(candidate, os.X_OK):
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def is_ccproxy_available() -> bool:
|
||||
"""Check whether the ``ccproxy`` CLI binary is available."""
|
||||
return _ccproxy_exe() is not None
|
||||
|
||||
|
||||
def _is_editable_install() -> bool:
|
||||
"""Return True if EvoScientist was installed in editable/development mode.
|
||||
|
||||
Checks all matching distributions because a stale ``.egg-info`` in the
|
||||
project root can shadow the real ``dist-info`` in site-packages.
|
||||
"""
|
||||
try:
|
||||
import importlib.metadata as _meta
|
||||
import json
|
||||
|
||||
for dist in _meta.distributions():
|
||||
name = dist.metadata.get("Name", "")
|
||||
if name.lower() != "evoscientist":
|
||||
continue
|
||||
direct_url = dist.read_text("direct_url.json")
|
||||
if direct_url is not None:
|
||||
data = json.loads(direct_url)
|
||||
if data.get("dir_info", {}).get("editable", False) is True:
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def _oauth_install_hint() -> str:
|
||||
"""Return the appropriate install command depending on install method."""
|
||||
if _is_editable_install():
|
||||
return "uv sync --extra oauth or pip install -e '.[oauth]'"
|
||||
return "pip install 'evoscientist[oauth]'"
|
||||
|
||||
|
||||
def _summarize_auth_output(raw: str) -> str:
|
||||
"""Extract key fields from ccproxy auth status output into a one-line summary.
|
||||
|
||||
Parses the Rich table output for Email, Subscription, and Status fields.
|
||||
Returns e.g. ``"user@example.com (plus, active)"``.
|
||||
Falls back to ``"Authenticated"`` if parsing fails.
|
||||
"""
|
||||
import re as _re
|
||||
|
||||
# Strip ANSI escape sequences
|
||||
clean = _re.sub(r"\x1b\[[0-9;]*m", "", raw)
|
||||
|
||||
# Parse "Key<2+ spaces>Value" table rows, match exact key names
|
||||
fields: dict[str, str] = {}
|
||||
for line in clean.splitlines():
|
||||
m = _re.match(r"\s*(.+?)\s{2,}(.+)", line)
|
||||
if not m:
|
||||
continue
|
||||
key, val = m.group(1).strip(), m.group(2).strip()
|
||||
if key in ("Email", "Subscription", "Subscription Status"):
|
||||
fields[key.lower().replace(" ", "_")] = val
|
||||
|
||||
email = fields.get("email", "")
|
||||
sub = fields.get("subscription", "")
|
||||
status = fields.get("subscription_status", "")
|
||||
|
||||
if email:
|
||||
detail = ", ".join(filter(None, [sub, status]))
|
||||
return f"{email} ({detail})" if detail else email
|
||||
return "Authenticated"
|
||||
|
||||
|
||||
def check_ccproxy_auth(provider: str = "claude_api") -> tuple[bool, str]:
|
||||
"""Check if ccproxy has valid OAuth credentials.
|
||||
|
||||
Args:
|
||||
provider: ccproxy provider name ("claude_api" or "codex").
|
||||
|
||||
Returns:
|
||||
(is_valid, message) tuple.
|
||||
"""
|
||||
try:
|
||||
exe = _ccproxy_exe() or "ccproxy"
|
||||
result = subprocess.run(
|
||||
[exe, "auth", "status", provider],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
import re as _re
|
||||
|
||||
raw = (result.stdout + result.stderr).strip()
|
||||
clean = _re.sub(r"\x1b\[[0-9;]*m", "", raw)
|
||||
|
||||
# Filter out structlog warning/noise lines, keep only status lines
|
||||
status_lines = [
|
||||
line
|
||||
for line in clean.splitlines()
|
||||
if line.strip()
|
||||
and not _re.match(r"\d{4}-\d{2}-\d{2}", line.strip())
|
||||
and "warning" not in line.lower()
|
||||
and "plugin" not in line.lower()
|
||||
]
|
||||
status_msg = " ".join(status_lines).strip()
|
||||
|
||||
# ccproxy auth status may exit 0 even when not authenticated —
|
||||
# detect failure by checking output content
|
||||
if result.returncode != 0 or "not authenticated" in clean.lower():
|
||||
return False, status_msg or "Not authenticated"
|
||||
|
||||
summary = _summarize_auth_output(result.stdout)
|
||||
return True, summary or "Authenticated"
|
||||
except FileNotFoundError:
|
||||
return False, "ccproxy not found"
|
||||
except subprocess.TimeoutExpired:
|
||||
return False, "Auth check timed out"
|
||||
except Exception as exc:
|
||||
return False, f"Auth check failed: {exc}"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Process management
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def is_ccproxy_running(port: int) -> bool:
|
||||
"""Check if ccproxy is already serving on the given port."""
|
||||
import httpx
|
||||
|
||||
try:
|
||||
resp = httpx.get(f"http://127.0.0.1:{port}/health/live", timeout=2.0)
|
||||
return resp.status_code == 200
|
||||
except (httpx.ConnectError, httpx.TimeoutException, OSError):
|
||||
return False
|
||||
|
||||
|
||||
def start_ccproxy(port: int) -> subprocess.Popen:
|
||||
"""Start ccproxy serve as a background process.
|
||||
|
||||
Args:
|
||||
port: Port number for the proxy server.
|
||||
|
||||
Returns:
|
||||
The Popen handle for the ccproxy process.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If ccproxy fails to become healthy within 30 seconds.
|
||||
FileNotFoundError: If ccproxy binary is not found.
|
||||
"""
|
||||
exe = _ccproxy_exe() or "ccproxy"
|
||||
proc = subprocess.Popen(
|
||||
[exe, "serve", "--port", str(port)],
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
|
||||
# Wait for health (ccproxy can take up to ~11s on first start)
|
||||
deadline = time.monotonic() + 30
|
||||
while time.monotonic() < deadline:
|
||||
if proc.poll() is not None:
|
||||
raise RuntimeError(
|
||||
f"ccproxy exited immediately with code {proc.returncode}"
|
||||
)
|
||||
if is_ccproxy_running(port):
|
||||
return proc
|
||||
time.sleep(0.3)
|
||||
|
||||
# Timed out — clean up
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=3)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
raise RuntimeError("ccproxy did not become healthy within 30 seconds")
|
||||
|
||||
|
||||
def stop_ccproxy(proc: subprocess.Popen | None) -> None:
|
||||
"""Gracefully stop a ccproxy process.
|
||||
|
||||
Safe to call with None (no-op).
|
||||
"""
|
||||
if proc is None:
|
||||
return
|
||||
try:
|
||||
proc.terminate()
|
||||
proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait(timeout=2)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def ensure_ccproxy(port: int) -> subprocess.Popen | None:
|
||||
"""Ensure ccproxy is running — reuse existing or start new.
|
||||
|
||||
Returns:
|
||||
Popen handle if we started a new process, None if already running.
|
||||
"""
|
||||
if is_ccproxy_running(port):
|
||||
logger.debug("ccproxy already running on port %d", port)
|
||||
return None
|
||||
return start_ccproxy(port)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Environment setup
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def setup_ccproxy_env(port: int) -> None:
|
||||
"""Set environment variables for Anthropic ccproxy routing.
|
||||
|
||||
Force-sets ``ANTHROPIC_BASE_URL`` and ``ANTHROPIC_API_KEY`` so that
|
||||
downstream LangChain/Anthropic clients route through ccproxy.
|
||||
|
||||
Always overrides existing values — when this function is called,
|
||||
we've decided to use ccproxy, so env must point to it.
|
||||
"""
|
||||
os.environ["ANTHROPIC_BASE_URL"] = f"http://127.0.0.1:{port}/claude"
|
||||
os.environ["ANTHROPIC_API_KEY"] = "ccproxy-oauth"
|
||||
|
||||
|
||||
def setup_codex_env(port: int) -> None:
|
||||
"""Set environment variables for OpenAI/Codex ccproxy routing.
|
||||
|
||||
Force-sets ``OPENAI_BASE_URL`` and ``OPENAI_API_KEY`` so that
|
||||
downstream LangChain/OpenAI clients route through ccproxy's Codex
|
||||
endpoint.
|
||||
|
||||
Always overrides existing values — when this function is called,
|
||||
we've decided to use ccproxy, so env must point to it.
|
||||
"""
|
||||
os.environ["OPENAI_BASE_URL"] = f"http://127.0.0.1:{port}/codex/v1"
|
||||
os.environ["OPENAI_API_KEY"] = "ccproxy-oauth"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# High-level orchestration
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def maybe_start_ccproxy(config: EvoScientistConfig) -> subprocess.Popen | None:
|
||||
"""High-level: conditionally start ccproxy based on config.
|
||||
|
||||
Checks ``config.anthropic_auth_mode`` and ``config.openai_auth_mode``:
|
||||
- ``oauth``: ccproxy must work — raises on failure.
|
||||
- ``api_key``: no-op for that provider.
|
||||
|
||||
When either provider uses OAuth, ccproxy is started (single process
|
||||
serves both providers). Environment variables are set for each
|
||||
provider that uses OAuth.
|
||||
|
||||
Args:
|
||||
config: An ``EvoScientistConfig`` instance.
|
||||
|
||||
Returns:
|
||||
Popen handle if we started ccproxy, None otherwise.
|
||||
"""
|
||||
anthropic_oauth = getattr(config, "anthropic_auth_mode", "api_key") == "oauth"
|
||||
openai_oauth = getattr(config, "openai_auth_mode", "api_key") == "oauth"
|
||||
|
||||
if not anthropic_oauth and not openai_oauth:
|
||||
return None
|
||||
|
||||
if not is_ccproxy_available():
|
||||
raise RuntimeError(
|
||||
"ccproxy is required for OAuth mode but not found. "
|
||||
f"Install it with: {_oauth_install_hint()}"
|
||||
)
|
||||
|
||||
# Check auth for each provider that uses OAuth
|
||||
if anthropic_oauth:
|
||||
authed, msg = check_ccproxy_auth("claude_api")
|
||||
if not authed:
|
||||
raise RuntimeError(
|
||||
f"ccproxy Anthropic OAuth not authenticated: {msg}\n"
|
||||
"Run: ccproxy auth login claude_api"
|
||||
)
|
||||
|
||||
if openai_oauth:
|
||||
authed, msg = check_ccproxy_auth("codex")
|
||||
if not authed:
|
||||
raise RuntimeError(
|
||||
f"ccproxy Codex OAuth not authenticated: {msg}\n"
|
||||
"Run: ccproxy auth login codex"
|
||||
)
|
||||
|
||||
port = config.ccproxy_port
|
||||
if not (1 <= port <= 65535):
|
||||
raise ValueError(f"Invalid ccproxy port: {port}. Must be between 1 and 65535.")
|
||||
|
||||
# Start ccproxy (single process serves both providers)
|
||||
proc = ensure_ccproxy(port)
|
||||
|
||||
# Set environment for each OAuth provider
|
||||
if anthropic_oauth:
|
||||
setup_ccproxy_env(port)
|
||||
if openai_oauth:
|
||||
setup_codex_env(port)
|
||||
|
||||
if proc:
|
||||
logger.info("Started ccproxy on port %d", port)
|
||||
else:
|
||||
logger.info("Reusing existing ccproxy on port %d", port)
|
||||
return proc
|
||||
@@ -0,0 +1,928 @@
|
||||
# Channels
|
||||
|
||||
EvoScientist provides unified integration with 10 messaging platforms. This document covers the architecture overview, message processing pipeline, capability matrix, security model, deployment guides, and troubleshooting.
|
||||
|
||||
Configuration file: `~/.config/ai4scientist/settings.yaml` (or use environment variables with the `EVOSCIENTIST_` prefix).
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Architecture](#architecture)
|
||||
- [Message Processing Pipeline](#message-processing-pipeline)
|
||||
- [Middleware Pipeline](#middleware-pipeline)
|
||||
- [Capability Matrix](#capability-matrix)
|
||||
- [Security and Access Control](#security-and-access-control)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Channel Deployment Guides](#channel-deployment-guides)
|
||||
- [Telegram](#telegram) | [Discord](#discord) | [Slack](#slack) | [Feishu (Lark)](#feishu-lark) | [WeChat](#wechat)
|
||||
- [DingTalk](#dingtalk) | [QQ](#qq) | [Signal](#signal) | [Email](#email) | [iMessage](#imessage)
|
||||
- [Running Multiple Channels](#running-multiple-channels)
|
||||
- [Docker Deployment](#docker-deployment)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Messaging Platforms │
|
||||
│ │
|
||||
│ ┌────────┐ ┌────────┐ ┌────────┐ │
|
||||
│ │Telegram│ │Discord │ │ Slack │ ...x10 │
|
||||
│ └───┬────┘ └───┬────┘ └───┬────┘ │
|
||||
└──────┼──────────┼──────────┼───────────────┘
|
||||
│ │ │
|
||||
┌──────┴──────────┴──────────┴───────────────┐
|
||||
│ Inbound Middleware │
|
||||
│ │
|
||||
│ Dedup → AllowList → Pairing → GroupHist │
|
||||
│ → Mention │
|
||||
└──────────────────┬─────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────┐
|
||||
│ MessageBus │
|
||||
│ │
|
||||
│ inbound queue ──► outbound queue │
|
||||
│ (asyncio.Queue, capacity 5000) │
|
||||
└──────────┬───────────────┬───────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────┐ ┌─────────────────┐
|
||||
│ InboundConsumer │ │ Dispatcher │
|
||||
│ │ │ │
|
||||
│ Worker pool (8) │ │ Routes replies │
|
||||
│ Per-chat locks │ │ to origin │
|
||||
│ Session dedup │ │ channel │
|
||||
│ Timeout handling │ │ │
|
||||
│ │ │ └─────────────────┘
|
||||
│ ▼ │
|
||||
│ Agent Core │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
### Core Modules
|
||||
|
||||
| Module | Responsibility |
|
||||
|--------|---------------|
|
||||
| `base.py` | Abstract `Channel` base class — readiness checks, retry strategy, mention stripping, format fallback, media handling, debounce, send locks |
|
||||
| `capabilities.py` | `ChannelCapabilities` frozen dataclass — each channel declares features, framework adapts automatically |
|
||||
| `plugin.py` | `ChannelPlugin` base with adapter slots — `ConfigAdapter`, `SecurityAdapter`, `GroupAdapter`, `MentionAdapter`, `OutboundAdapter`, `ThreadingAdapter`, etc. |
|
||||
| `mixins.py` | Reusable async patterns: `WebhookMixin` (aiohttp server + httpx client), `WebSocketMixin` (connect/reconnect/heartbeat), `PollingMixin` (async polling loop), `TokenMixin` (OAuth token auto-refresh) |
|
||||
| `config.py` | `BaseChannelConfig` — shared config fields (allowed_senders, proxy, text_chunk_limit, etc.) + `SingleAccountConfigAdapter` / `MultiAccountConfigAdapter` |
|
||||
| `bus/` | `MessageBus` async event bus with `InboundMessage` / `OutboundMessage` dataclasses, decoupling channels from agent core |
|
||||
| `channel_manager.py` | `ChannelManager` — lifecycle management (start/stop), health monitoring, channel registry, account management, outbound dispatch |
|
||||
| `consumer.py` | `InboundConsumer` — worker pool, per-chat serial locks, session deduplication, timeout handling |
|
||||
| `retry.py` | `RetryConfig` — exponential backoff retry with per-channel presets (attempts, min/max delay, jitter) |
|
||||
| `formatter.py` | `UnifiedFormatter` — Markdown to platform-specific format conversion (HTML, Slack mrkdwn, Discord, plain text) |
|
||||
| `standalone.py` | Headless channel runner (`run_standalone`) for running channels without the CLI |
|
||||
|
||||
## Message Processing Pipeline
|
||||
|
||||
### Inbound (User Message → Agent)
|
||||
|
||||
```
|
||||
1. Platform SDK/Webhook receives raw message
|
||||
│
|
||||
2. Channel._on_message() parses into RawIncoming
|
||||
│
|
||||
3. Channel._enqueue_raw() runs middleware pipeline:
|
||||
├── DedupMiddleware — drop duplicates (LRU cache, 60s TTL)
|
||||
├── AllowListMiddleware — enforce sender/channel restrictions
|
||||
├── PairingMiddleware — handle DM pairing flow (if dm_policy="pairing")
|
||||
├── GroupHistoryMiddleware — buffer group context for injection
|
||||
└── MentionGatingMiddleware — filter by @mention policy in groups
|
||||
│
|
||||
4. InboundMessage queued on Channel._queue
|
||||
│
|
||||
5. Channel.run() → receive() → queue_message() with debounce
|
||||
│
|
||||
│ (500ms debounce window: rapid messages from same sender merged)
|
||||
│
|
||||
6. MessageBus.publish_inbound()
|
||||
│
|
||||
7. InboundConsumer acquires per-chat lock → invokes Agent
|
||||
│
|
||||
8. Agent response → OutboundMessage → MessageBus.publish_outbound()
|
||||
```
|
||||
|
||||
### Outbound (Agent Response → User)
|
||||
|
||||
```
|
||||
1. OutboundMessage arrives on MessageBus outbound queue
|
||||
│
|
||||
2. Dispatcher routes to origin channel by name
|
||||
│
|
||||
3. Channel.send() processes the response:
|
||||
├── Stop typing indicator
|
||||
├── Format text (Markdown → platform format)
|
||||
├── Chunk text to platform limit (code-block-aware splitting)
|
||||
├── Send each chunk via _send_chunk() with format fallback
|
||||
├── Send media attachments via _send_media_impl()
|
||||
└── Retry on transient errors (exponential backoff)
|
||||
```
|
||||
|
||||
### Text Chunking
|
||||
|
||||
Long responses are split intelligently with this priority:
|
||||
|
||||
1. Markdown code block fence boundaries
|
||||
2. Double newlines (paragraph breaks)
|
||||
3. Single newlines
|
||||
4. Space characters
|
||||
5. Hard cut at limit (last resort)
|
||||
|
||||
Code blocks are never split mid-block when possible. Each chunk is sent as a separate message.
|
||||
|
||||
## Middleware Pipeline
|
||||
|
||||
Middleware runs sequentially on each inbound message. Each middleware can pass, modify, or drop the message.
|
||||
|
||||
### 1. DedupMiddleware
|
||||
|
||||
Prevents duplicate message processing using a bounded LRU cache with TTL.
|
||||
|
||||
- Cache size: 1000 entries (configurable)
|
||||
- TTL: 60 seconds
|
||||
- Key: `message_id` from the platform
|
||||
- Messages with the same ID within the TTL window are silently dropped
|
||||
|
||||
### 2. AllowListMiddleware
|
||||
|
||||
Enforces sender and channel restrictions based on the `dm_policy` config.
|
||||
|
||||
| Policy | Behavior |
|
||||
|--------|----------|
|
||||
| `"open"` | Accept messages from anyone |
|
||||
| `"allowlist"` | Only accept from `allowed_senders` / `allowed_channels` |
|
||||
| `"pairing"` | Require DM pairing before accepting (see PairingMiddleware) |
|
||||
|
||||
When `allowed_senders` is set (non-empty), only messages from listed sender IDs pass through. Same for `allowed_channels`.
|
||||
|
||||
### 3. PairingMiddleware
|
||||
|
||||
Handles an interactive DM pairing flow for the `"pairing"` dm_policy.
|
||||
|
||||
- First message from an unknown sender triggers a pairing request
|
||||
- The sender must provide a valid pairing code
|
||||
- Once paired, the sender is added to the allowlist for future messages
|
||||
|
||||
### 4. GroupHistoryMiddleware
|
||||
|
||||
Buffers recent group chat messages to provide conversation context.
|
||||
|
||||
- Only active when `capabilities.groups = True`
|
||||
- Maintains a per-chat rolling buffer (default: 50 messages, 5-minute max age)
|
||||
- When the bot is mentioned in a group, recent history is injected into the message metadata so the agent can see prior context
|
||||
- Non-mentioned group messages are buffered but not forwarded (see MentionGating)
|
||||
|
||||
### 5. MentionGatingMiddleware
|
||||
|
||||
Controls whether the bot responds in group chats.
|
||||
|
||||
| `require_mention` | Behavior |
|
||||
|-------------------|----------|
|
||||
| `True` / `"group"` | Only respond to @mentions in groups; always respond in DMs |
|
||||
| `False` / `"none"` | Respond to all messages in all contexts |
|
||||
| `"always"` | Require @mention even in DMs |
|
||||
|
||||
Default: `"group"` — the bot ignores group messages unless explicitly @mentioned.
|
||||
|
||||
Mention detection is platform-specific:
|
||||
- **Telegram**: checks for `@bot_username` in text
|
||||
- **Discord**: checks `message.mentions` for bot user
|
||||
- **Slack**: handled via separate `app_mention` event type
|
||||
- **Feishu**: checks `mentions` array in event payload
|
||||
- **DingTalk**: checks `isInAtList` flag or `atUsers` array
|
||||
- **WeChat (WeCom)**: checks `AtUserList` XML field
|
||||
|
||||
## Capability Matrix
|
||||
|
||||
| Channel | Format | Max Len | Media | Voice | Sticker | Location | Video | Typing | Reaction | Thread | Group | @Mention | No Public IP | Token Refresh | Proxy | Allowlist |
|
||||
|:--------|:------:|:-------:|:-----:|:-----:|:-------:|:--------:|:-----:|:------:|:--------:|:------:|:-----:|:--------:|:------------:|:-------------:|:-----:|:---------:|
|
||||
| Telegram | HTML | 4000 | S/R | R | R | R | | 4s | emoji | | G | @ | yes | | yes | yes |
|
||||
| Discord | Discord | 2000 | S/R | | | | | 8s | emoji | yes | G | @ | yes | | yes | yes |
|
||||
| Slack | Mrkdwn | 4000 | S/R | | | | | post | emoji | yes | G | @ | yes | | yes | yes |
|
||||
| Feishu | Post | 4096 | S/R | R | R | | | | emoji | | G | @ | ws mode | 2h | yes | yes |
|
||||
| WeChat | MD | 4096 | S/R | R | | R | R | recall | | | G | @ | no | 2h | yes | yes |
|
||||
| DingTalk | MD | 4096 | S/R | R | | | R | | | | G | @ | yes | 2h | yes | yes |
|
||||
| QQ | MD/plain | 4096 | S/R | | | | | | | | G | @ | yes | | | yes |
|
||||
| Signal | Plain | 4096 | S/R | R | | | | api | emoji | | G | UUID | yes | | | yes |
|
||||
| iMessage | Plain | - | S/R | R | | | | | | | G | | yes | | | yes |
|
||||
| Email | HTML | - | S/R | | | | | | | | | | yes | | | yes |
|
||||
|
||||
Legend: **S** = send, **R** = receive, **G** = group chat, **@** = @mention detection, **-** = no practical limit
|
||||
|
||||
### Connection Types
|
||||
|
||||
| Channel | Transport | Connection Mode | Default Port |
|
||||
|-------------|-----------|----------------------------------------|:------------:|
|
||||
| Telegram | HTTPS | Long polling (`getUpdates`) | -- |
|
||||
| Discord | WebSocket | Gateway events (`discord.py`) | -- |
|
||||
| Slack | WebSocket | Socket Mode (`slack-sdk`) | -- |
|
||||
| Feishu | HTTP/WS | Webhook or WebSocket long connection | 9000/-- |
|
||||
| WeChat | HTTP | Webhook `POST /wechat/callback` | 9001 |
|
||||
| DingTalk | WebSocket | Stream Mode (DingTalk gateway) | -- |
|
||||
| QQ | WebSocket | Bot Gateway (`qq-botpy`) | -- |
|
||||
| Signal | TCP | JSON-RPC (`signal-cli` daemon) | 7583 |
|
||||
| iMessage | stdio | JSON-RPC (`imsg` CLI) | -- |
|
||||
| Email | TCP | IMAP polling + SMTP send | 993/587 |
|
||||
|
||||
> **"--"** means no listening port is required -- no public IP or port forwarding needed.
|
||||
|
||||
### Format Conversion
|
||||
|
||||
The `UnifiedFormatter` converts Markdown output from the agent into platform-native formats:
|
||||
|
||||
| Target Format | Conversion |
|
||||
|:-------------|:-----------|
|
||||
| HTML (Telegram, Email) | `**bold**` → `<b>bold</b>`, `` `code` `` → `<code>code</code>`, code blocks → `<pre>`, special chars escaped |
|
||||
| Slack mrkdwn | `**bold**` → `*bold*`, `_italic_` → `_italic_`, code blocks preserved, `<>&` escaped |
|
||||
| Discord Markdown | Mostly passthrough, minor adjustments for Discord-specific rendering |
|
||||
| Feishu Post | Markdown → Feishu rich text JSON (code blocks, bold, italic, strikethrough, links, headings, quotes, lists) |
|
||||
| Plain text | All formatting stripped, structure preserved via indentation |
|
||||
|
||||
## Security and Access Control
|
||||
|
||||
### Sender Allowlist
|
||||
|
||||
Every channel supports `allowed_senders` to restrict who can interact with the bot:
|
||||
|
||||
```yaml
|
||||
telegram_allowed_senders: "123456789,987654321" # Telegram user IDs
|
||||
discord_allowed_senders: "111222333444555666" # Discord user IDs
|
||||
slack_allowed_senders: "U0123ABCDEF" # Slack Member IDs
|
||||
feishu_allowed_senders: "ou_xxxxxxxxxxxx" # Feishu open_ids
|
||||
signal_allowed_senders: "+1234567890" # Phone numbers
|
||||
email_allowed_senders: "alice@example.com" # Email addresses
|
||||
imessage_allowed_senders: "+1234567890,user@icloud.com" # Phone or email
|
||||
```
|
||||
|
||||
When `allowed_senders` is empty, the channel accepts messages from anyone. **For production deployments, always set an allowlist.**
|
||||
|
||||
### Channel Allowlist
|
||||
|
||||
For platforms with multiple channels/groups (Discord, Slack), restrict which channels the bot operates in:
|
||||
|
||||
```yaml
|
||||
discord_allowed_channels: "111222333444555666,777888999000111222"
|
||||
slack_allowed_channels: "C0123ABCDEF,C0456GHIJKL"
|
||||
```
|
||||
|
||||
### Token and Secret Handling
|
||||
|
||||
- API tokens are stored in the config file or environment variables, never logged at INFO level
|
||||
- Discord logs only the first 8 and last 4 characters of the bot token for debugging
|
||||
- WeChat/Feishu tokens are auto-refreshed before expiry (5-minute margin on 2-hour TTL)
|
||||
- Webhook signature verification is enforced when `token`/`encoding_aes_key` is configured (WeChat, Feishu)
|
||||
|
||||
### Group Chat Behavior
|
||||
|
||||
By default, the bot only responds in group chats when explicitly @mentioned. This prevents the bot from responding to every message in a busy group. Configure via:
|
||||
|
||||
```yaml
|
||||
# Default: only respond when mentioned in groups
|
||||
channel_require_mention: "group"
|
||||
|
||||
# Respond to all messages (including groups)
|
||||
channel_require_mention: "none"
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Install channel dependencies
|
||||
|
||||
```bash
|
||||
pip install evoscientist[telegram]
|
||||
# Available extras: telegram, discord, slack, feishu, wechat,
|
||||
# dingtalk, qq, email, signal
|
||||
# iMessage requires no extra Python dependencies
|
||||
```
|
||||
|
||||
### 2. Configure
|
||||
|
||||
```bash
|
||||
# Option A: Interactive wizard
|
||||
EvoSci onboard
|
||||
|
||||
# Option B: CLI commands
|
||||
EvoSci config set channel_enabled telegram
|
||||
EvoSci config set telegram_bot_token "123456:ABC-xxx"
|
||||
|
||||
# Option C: Environment variables (EVOSCIENTIST_ prefix, uppercase)
|
||||
export EVOSCIENTIST_CHANNEL_ENABLED=telegram
|
||||
export EVOSCIENTIST_TELEGRAM_BOT_TOKEN="123456:ABC-xxx"
|
||||
```
|
||||
|
||||
### 3. Start
|
||||
|
||||
```bash
|
||||
EvoSci serve # Start agent + all enabled channels
|
||||
# or
|
||||
EvoSci channel start # Standalone channel mode (message loop only)
|
||||
```
|
||||
|
||||
### 4. Health check
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/healthz
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"channels": { "enabled": ["telegram"], "running": ["telegram"] }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Channel Deployment Guides
|
||||
|
||||
---
|
||||
|
||||
### Telegram
|
||||
|
||||
**Install:** `pip install evoscientist[telegram]`
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Search for [@BotFather](https://t.me/BotFather) in Telegram, send `/newbot`, and follow the prompts to create a bot.
|
||||
2. BotFather will return a Bot Token (format: `123456789:ABCdefGHI...`) -- save it securely.
|
||||
3. Get your user ID: send any message to [@userinfobot](https://t.me/userinfobot), it will reply with your numeric ID.
|
||||
4. (Optional) For group use: add the bot to a group, then in BotFather send `/setprivacy` -> `Disable` so the bot can read group messages.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
channel_enabled: "telegram"
|
||||
telegram_bot_token: "123456789:ABCdefGHIjklMNOpqrSTUvwxYZ"
|
||||
telegram_allowed_senders: "" # Comma-separated user IDs; empty = no restriction
|
||||
telegram_proxy: "" # Optional HTTPS proxy (e.g. http://proxy:8080)
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `telegram_bot_token` | `str` | `""` | **Required.** Bot API Token from BotFather |
|
||||
| `telegram_allowed_senders` | `str` | `""` | Comma-separated user IDs, empty = allow all |
|
||||
| `telegram_proxy` | `str` | `""` | HTTPS proxy URL |
|
||||
|
||||
**Env vars:** `EVOSCIENTIST_TELEGRAM_BOT_TOKEN`, `EVOSCIENTIST_TELEGRAM_ALLOWED_SENDERS`, `EVOSCIENTIST_TELEGRAM_PROXY`
|
||||
|
||||
**Technical details:** Long polling mode, `drop_pending_updates=True` on startup to skip backlog. Markdown to Telegram HTML auto-conversion (bold, italic, strikethrough, links, code blocks, headings, lists). Falls back to plain text on HTML parse failure. Media routed by extension to `send_photo`/`send_video`/`send_audio`/`send_document`. In groups, only responds when @mentioned; auto-strips @mention. Typing indicator refreshes every 4s. ACK reaction (eyes emoji) on message receipt, removed after reply. Retry: 3 attempts, min delay 0.4s, parse errors not retried. Text chunk limit: 4000 chars.
|
||||
|
||||
---
|
||||
|
||||
### Discord
|
||||
|
||||
**Install:** `pip install evoscientist[discord]`
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Go to [Discord Developer Portal](https://discord.com/developers/applications) -> New Application -> enter a name.
|
||||
2. Left menu **Bot** -> Reset Token -> copy the Bot Token.
|
||||
3. Under **Privileged Gateway Intents**, enable **Message Content Intent** (required to read message content).
|
||||
4. Left menu **OAuth2** -> URL Generator:
|
||||
- Scopes: check `bot`
|
||||
- Bot Permissions: check `Send Messages`, `Read Message History`, `Attach Files`, `Add Reactions`
|
||||
- Copy the generated URL, open in browser, select a server to invite the bot.
|
||||
5. Get user ID: Discord Settings -> Advanced -> enable Developer Mode -> right-click username -> Copy User ID.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
channel_enabled: "discord"
|
||||
discord_bot_token: "MTIzNDU2Nzg5.xxxx.xxxxx"
|
||||
discord_allowed_senders: "" # Comma-separated user IDs
|
||||
discord_allowed_channels: "" # Comma-separated channel IDs
|
||||
discord_proxy: ""
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `discord_bot_token` | `str` | `""` | **Required.** Bot Token |
|
||||
| `discord_allowed_senders` | `str` | `""` | Comma-separated user IDs, empty = allow all |
|
||||
| `discord_allowed_channels` | `str` | `""` | Comma-separated channel IDs, empty = allow all |
|
||||
| `discord_proxy` | `str` | `""` | HTTPS proxy URL |
|
||||
|
||||
**Env vars:** `EVOSCIENTIST_DISCORD_BOT_TOKEN`, `EVOSCIENTIST_DISCORD_ALLOWED_SENDERS`, `EVOSCIENTIST_DISCORD_ALLOWED_CHANNELS`, `EVOSCIENTIST_DISCORD_PROXY`
|
||||
|
||||
**Technical details:** WebSocket Gateway (`discord.py`). In server channels, only responds when @mentioned; DMs respond directly. Thread-aware: messages in threads are tracked with `parent_channel_id` and `thread_id`. Replies via `MessageReference`. Message cache (200 entries) for ACK emoji reactions. Attachment download (max 20 MB) with safe filename sanitization. Media sent via `discord.File`. Typing indicator refreshes every 8s. Retry: 3 attempts, parses `Retry-After` header for 429s. Text chunk limit: 2000 chars.
|
||||
|
||||
---
|
||||
|
||||
### Slack
|
||||
|
||||
**Install:** `pip install evoscientist[slack]`
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Go to [Slack API](https://api.slack.com/apps) -> Create New App -> From scratch -> select workspace.
|
||||
2. Left menu **Socket Mode** -> enable -> Generate App-Level Token, scope `connections:write` -> copy App Token (`xapp-...`).
|
||||
3. Left menu **OAuth & Permissions** -> add Bot Token Scopes:
|
||||
- `chat:write`, `channels:history`, `groups:history`, `im:history`, `files:read`, `files:write`, `reactions:write`
|
||||
4. Click **Install to Workspace** -> copy Bot User OAuth Token (`xoxb-...`).
|
||||
5. Left menu **Event Subscriptions** -> enable -> Subscribe to bot events: `message.channels`, `message.groups`, `message.im`, `app_mention`.
|
||||
6. Get Member ID: click user avatar -> profile -> **...** -> Copy member ID.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
channel_enabled: "slack"
|
||||
slack_bot_token: "xoxb-xxxx-xxxx-xxxx"
|
||||
slack_app_token: "xapp-1-xxxx-xxxx"
|
||||
slack_allowed_senders: "" # Member ID (U...)
|
||||
slack_allowed_channels: "" # Channel ID (C...)
|
||||
slack_proxy: ""
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `slack_bot_token` | `str` | `""` | **Required.** Bot User OAuth Token (`xoxb-`) |
|
||||
| `slack_app_token` | `str` | `""` | **Required.** Socket Mode App Token (`xapp-`) |
|
||||
| `slack_allowed_senders` | `str` | `""` | Comma-separated Member IDs |
|
||||
| `slack_allowed_channels` | `str` | `""` | Comma-separated Channel IDs |
|
||||
| `slack_proxy` | `str` | `""` | HTTPS proxy URL |
|
||||
|
||||
**Env vars:** `EVOSCIENTIST_SLACK_BOT_TOKEN`, `EVOSCIENTIST_SLACK_APP_TOKEN`, `EVOSCIENTIST_SLACK_ALLOWED_SENDERS`, `EVOSCIENTIST_SLACK_ALLOWED_CHANNELS`, `EVOSCIENTIST_SLACK_PROXY`
|
||||
|
||||
**Technical details:** Socket Mode (no public URL needed). Markdown to mrkdwn conversion. DMs respond directly; channels respond to `app_mention` events. Thread replies via `thread_ts` -- all replies are threaded to the original message. Typing indicator approximated by posting/deleting a "..." message (Slack has no bot typing API). ACK reaction (eyes emoji) on message receipt. Attachments downloaded with Bearer auth. Media sent via `files_upload_v2`. Runs `auth_test()` on startup to verify credentials and cache bot user ID. Retry: 3 attempts, exponential backoff + jitter. Text chunk limit: 4000 chars.
|
||||
|
||||
---
|
||||
|
||||
### Feishu (Lark)
|
||||
|
||||
**Install:** `pip install evoscientist[feishu]`
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Go to [Feishu Open Platform](https://open.feishu.cn/app) (international: [Lark Developer](https://open.larksuite.com/app)) -> create a custom app.
|
||||
2. Copy the **App ID** and **App Secret**.
|
||||
3. Left menu **Event Subscriptions**:
|
||||
- **Webhook mode**: set request URL to `http://your-host:9000/webhook/event` -> copy **Verification Token** and **Encrypt Key**.
|
||||
- **WebSocket mode**: select **长连接** (Long Connection) as the subscription method. No URL needed.
|
||||
4. Add event: `im.message.receive_v1` (receive messages).
|
||||
5. Left menu **Permissions** -> enable `im:message:send_as_bot`.
|
||||
6. Create a version and publish.
|
||||
|
||||
> **Webhook mode** requires a publicly reachable URL. For local dev, use `ngrok http 9000` or [natapp](https://natapp.cn/) (recommended for China).
|
||||
|
||||
#### Subscription Modes
|
||||
|
||||
Feishu supports two subscription modes:
|
||||
|
||||
| Mode | Transport | Public IP? | Best For |
|
||||
|------|-----------|:----------:|----------|
|
||||
| `webhook` (default) | HTTP POST callback | Yes | Servers with public IP / cloud deployment |
|
||||
| `websocket` | WebSocket long connection | **No** | Local dev, behind NAT/firewall, China (no ngrok needed) |
|
||||
|
||||
**WebSocket mode** uses the official `lark-oapi` SDK to maintain an outbound WebSocket connection to Feishu servers. No public IP, port forwarding, or tunnel is required.
|
||||
|
||||
To use WebSocket mode:
|
||||
|
||||
```bash
|
||||
# Install the SDK
|
||||
pip install 'evoscientist[feishu]'
|
||||
|
||||
# Via config file
|
||||
feishu_subscription_mode: "websocket"
|
||||
|
||||
# Via CLI (standalone)
|
||||
python -m EvoScientist.channels.feishu.serve --app-id ID --app-secret SECRET --mode websocket
|
||||
```
|
||||
|
||||
> **Note:** In WebSocket mode, `feishu_verification_token`, `feishu_encrypt_key`, and `feishu_webhook_port` are not used — the SDK handles authentication and encryption internally.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
channel_enabled: "feishu"
|
||||
feishu_app_id: "cli_xxxxxxx"
|
||||
feishu_app_secret: "xxxxxxxxxxxxxxxxxx"
|
||||
feishu_subscription_mode: "webhook" # or "websocket"
|
||||
feishu_webhook_port: 9000
|
||||
feishu_allowed_senders: "" # open_id
|
||||
feishu_domain: "https://open.feishu.cn"
|
||||
feishu_proxy: ""
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `feishu_app_id` | `str` | `""` | **Required.** App ID |
|
||||
| `feishu_app_secret` | `str` | `""` | **Required.** App Secret |
|
||||
| `feishu_webhook_port` | `int` | `9000` | Webhook HTTP port |
|
||||
| `feishu_allowed_senders` | `str` | `""` | Comma-separated open_ids |
|
||||
| `feishu_domain` | `str` | `"https://open.feishu.cn"` | API domain (use `https://open.larksuite.com` for Lark) |
|
||||
| `feishu_proxy` | `str` | `""` | HTTPS proxy URL |
|
||||
| `feishu_subscription_mode` | `str` | `"webhook"` | `"webhook"` or `"websocket"` (WebSocket long connection, no public IP) |
|
||||
|
||||
**Env vars:** `EVOSCIENTIST_FEISHU_APP_ID`, `EVOSCIENTIST_FEISHU_APP_SECRET`, `EVOSCIENTIST_FEISHU_WEBHOOK_PORT`, `EVOSCIENTIST_FEISHU_DOMAIN`
|
||||
|
||||
**Technical details:** Webhook on `POST /webhook/event` with URL verification challenge-response. Supports both v1 (legacy) and v2 event schemas. Optional AES-256-CBC event decryption (when `encrypt_key` configured). `tenant_access_token` auto-refresh (2h TTL, refreshes 5 min before expiry). Markdown to Feishu Post rich text conversion (code blocks, bold, italic, strikethrough, links, headings, quotes, ordered/unordered lists). Plain text fallback. Group @mention filtering with mention key caching. Media: images via `/im/v1/images`, files via `/im/v1/files`. Replies via `/messages/{id}/reply` API. ACK reaction via `/messages/{id}/reactions`. Retry: 3 attempts, rate limit delay 2.0s, matches `99991400`/`rate limit`. Non-retryable: permission denied (`99991401`), invalid credentials. Text chunk limit: 4096 chars.
|
||||
|
||||
---
|
||||
|
||||
### WeChat
|
||||
|
||||
**Install:** `pip install evoscientist[wechat]`
|
||||
|
||||
Two backends supported: **WeCom** (recommended, free, no certification needed) and **WeChat Official Account** (requires verified service account).
|
||||
|
||||
#### WeCom
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Log in to [WeCom Admin Console](https://work.weixin.qq.com) -> App Management -> create a custom app.
|
||||
2. Copy the **Corp ID**, **AgentId**, and **Secret**.
|
||||
3. In app details -> Receive Messages -> Set API Receive -> URL: `http://your-host:9001/wechat/callback` -> copy **Token** and **EncodingAESKey**.
|
||||
4. In app details -> **Trusted IP** -> add your server's public IP address. Without this, all API calls will fail with error `60020`.
|
||||
|
||||
```yaml
|
||||
channel_enabled: "wechat"
|
||||
wechat_backend: "wecom"
|
||||
wechat_webhook_port: 9001
|
||||
wechat_wecom_corp_id: "ww..."
|
||||
wechat_wecom_agent_id: "1000002"
|
||||
wechat_wecom_secret: "xxxxxxxxxxxxxxxxxx"
|
||||
wechat_wecom_token: "xxxxxxxxxxxxxxxxxx"
|
||||
wechat_wecom_encoding_aes_key: "xxxxxxxxxxxxxxxxxx"
|
||||
wechat_allowed_senders: ""
|
||||
wechat_proxy: ""
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `wechat_backend` | `str` | `"wecom"` | `"wecom"` or `"wechatmp"` |
|
||||
| `wechat_webhook_port` | `int` | `9001` | Callback HTTP port |
|
||||
| `wechat_wecom_corp_id` | `str` | `""` | **Required (WeCom).** Corp ID |
|
||||
| `wechat_wecom_agent_id` | `str` | `""` | **Required (WeCom).** App AgentId |
|
||||
| `wechat_wecom_secret` | `str` | `""` | **Required (WeCom).** App Secret |
|
||||
| `wechat_wecom_token` | `str` | `""` | **Required (WeCom).** Callback Token |
|
||||
| `wechat_wecom_encoding_aes_key` | `str` | `""` | **Required (WeCom).** Callback EncodingAESKey |
|
||||
|
||||
#### WeChat Official Account
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Log in to [WeChat Official Account Platform](https://mp.weixin.qq.com) -> Settings & Development -> Basic Configuration.
|
||||
2. Copy the **AppID** and **AppSecret**.
|
||||
3. Server Configuration -> URL: `http://your-host:9001/wechat/callback` -> set **Token** and **EncodingAESKey**.
|
||||
|
||||
```yaml
|
||||
wechat_backend: "wechatmp"
|
||||
wechat_mp_app_id: "wx..."
|
||||
wechat_mp_app_secret: "xxxxxxxxxxxxxxxxxx"
|
||||
wechat_mp_token: "xxxxxxxxxxxxxxxxxx"
|
||||
wechat_mp_encoding_aes_key: "xxxxxxxxxxxxxxxxxx"
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `wechat_mp_app_id` | `str` | `""` | **Required (MP).** AppID |
|
||||
| `wechat_mp_app_secret` | `str` | `""` | **Required (MP).** AppSecret |
|
||||
| `wechat_mp_token` | `str` | `""` | **Required (MP).** Server Token |
|
||||
| `wechat_mp_encoding_aes_key` | `str` | `""` | **Required (MP).** Server EncodingAESKey |
|
||||
|
||||
**Technical details:** Webhook HTTP server for inbound (XML message parsing). GET callback for URL verification (SHA1 signature check). POST callback for message handling -- returns `"success"` within 5s and processes asynchronously. Optional AES encryption/decryption via `WeChatCrypto`. `access_token` auto-refresh (2h TTL, 5-min margin). Token-expired errors (40014, 42001) trigger automatic retry with refreshed token. WeCom supports Markdown message format with plain text fallback; Official Account uses plain text only (customer service API). WeCom group messages sent via `/appchat/send` endpoint (group IDs start with `wr`). Typing indicator approximated by posting/recalling a "..." message (WeCom only). Supports text, image, voice, video, location, file, and link message types. Media upload via `/media/upload`. Text chunk limit: 4096 chars.
|
||||
|
||||
---
|
||||
|
||||
### DingTalk
|
||||
|
||||
**Install:** `pip install evoscientist[dingtalk]`
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Go to [DingTalk Open Platform](https://open-dev.dingtalk.com) -> App Development -> create a bot app.
|
||||
2. Copy the **AppKey** (Client ID) and **AppSecret** (Client Secret).
|
||||
3. Enable **Stream Mode** in the app configuration -- no public IP needed.
|
||||
4. Publish the app and add the bot to a group, or test via direct message.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
channel_enabled: "dingtalk"
|
||||
dingtalk_client_id: "ding..."
|
||||
dingtalk_client_secret: "xxxxxxxxxxxxxxxxxx"
|
||||
dingtalk_allowed_senders: ""
|
||||
dingtalk_proxy: ""
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `dingtalk_client_id` | `str` | `""` | **Required.** AppKey |
|
||||
| `dingtalk_client_secret` | `str` | `""` | **Required.** AppSecret |
|
||||
| `dingtalk_allowed_senders` | `str` | `""` | Comma-separated user IDs |
|
||||
| `dingtalk_proxy` | `str` | `""` | HTTPS proxy URL |
|
||||
|
||||
**Env vars:** `EVOSCIENTIST_DINGTALK_CLIENT_ID`, `EVOSCIENTIST_DINGTALK_CLIENT_SECRET`
|
||||
|
||||
**Technical details:** Stream Mode via WebSocket -- connects to DingTalk gateway (`/v1.0/gateway/connections/open`) with automatic ticket-based auth. Ping/pong heartbeat with system topic handling. Message ACK via JSON response. `accessToken` auto-refresh. Sends via robot `oToMessages/batchSend` API in Markdown format (`sampleMarkdown`). Image uploads via `/media/upload` API with `sampleImageMsg`. Group @mention detection via `isInAtList` flag with `atUsers` array fallback. `downloadCode` resolution via `/robot/messageFiles/download` API for file/image/video/audio attachments. Auth errors (`invalidauthentication`/`forbidden`/`40014`) not retried. Text chunk limit: 4096 chars.
|
||||
|
||||
---
|
||||
|
||||
### QQ
|
||||
|
||||
**Install:** `pip install evoscientist[qq]`
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Go to [QQ Open Platform](https://q.qq.com) -> create a bot application.
|
||||
2. Complete developer verification, create a sandbox or production bot.
|
||||
3. Copy the **AppID** and **AppSecret**.
|
||||
4. Search for and add the bot as a friend in QQ, or add it to a group.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
channel_enabled: "qq"
|
||||
qq_app_id: "xxxxxxxxxx"
|
||||
qq_app_secret: "xxxxxxxxxxxxxxxxxx"
|
||||
qq_allowed_senders: ""
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `qq_app_id` | `str` | `""` | **Required.** AppID |
|
||||
| `qq_app_secret` | `str` | `""` | **Required.** AppSecret |
|
||||
| `qq_allowed_senders` | `str` | `""` | Comma-separated user IDs |
|
||||
|
||||
**Env vars:** `EVOSCIENTIST_QQ_APP_ID`, `EVOSCIENTIST_QQ_APP_SECRET`
|
||||
|
||||
**Technical details:** Uses `qq-botpy` SDK via WebSocket to connect to QQ Bot Gateway. Supports C2C (direct) and group messages. Outbound replies prefer native QQ markdown messages (`msg_type=2`) so headings/lists/code fences keep their structure; when the SDK/API rejects markdown, EvoScientist falls back to plain text with Markdown stripped but line structure preserved. Message deduplication (1000-entry LRU cache). Group @mention filtering (strips first `@bot`). Intents: `public_messages=True`, `direct_message=True`. Text chunk limit: 4096 chars.
|
||||
|
||||
---
|
||||
|
||||
### Signal
|
||||
|
||||
**Install:** `pip install evoscientist[signal]` (also requires [signal-cli](https://github.com/AsamK/signal-cli) installed separately)
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Install signal-cli: see [signal-cli installation guide](https://github.com/AsamK/signal-cli#installation).
|
||||
2. Register or link a phone number:
|
||||
- Register: `signal-cli -u +1234567890 register`, then `signal-cli -u +1234567890 verify CODE`
|
||||
- Link existing device: `signal-cli link -n "EvoScientist"`
|
||||
3. EvoScientist will auto-start the signal-cli daemon if it's not already running.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
channel_enabled: "signal"
|
||||
signal_phone_number: "+1234567890"
|
||||
signal_cli_path: "signal-cli"
|
||||
signal_config_dir: ""
|
||||
signal_allowed_senders: ""
|
||||
signal_rpc_port: 7583
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `signal_phone_number` | `str` | `""` | **Required.** Signal phone number (E.164 format) |
|
||||
| `signal_cli_path` | `str` | `"signal-cli"` | Path to signal-cli binary |
|
||||
| `signal_config_dir` | `str` | `""` | signal-cli config directory (optional) |
|
||||
| `signal_allowed_senders` | `str` | `""` | Comma-separated phone numbers |
|
||||
| `signal_rpc_port` | `int` | `7583` | JSON RPC socket port |
|
||||
|
||||
**Env vars:** `EVOSCIENTIST_SIGNAL_PHONE_NUMBER`, `EVOSCIENTIST_SIGNAL_CLI_PATH`, `EVOSCIENTIST_SIGNAL_RPC_PORT`
|
||||
|
||||
**Technical details:** JSON RPC over TCP socket to signal-cli daemon. Auto-starts daemon if not running (`signal-cli -u +NUMBER daemon --socket localhost:PORT`). Listens for `receive` notifications. Sends via `send` RPC method. Group detection via `groupInfo`. Mention detection via UUID matching. No public IP needed. Text chunk limit: 4096 chars.
|
||||
|
||||
---
|
||||
|
||||
### Email
|
||||
|
||||
**Install:** `pip install evoscientist[email]` (core dependencies included, no extras needed)
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Prepare an email account with IMAP + SMTP support (Gmail, Outlook, self-hosted, etc.).
|
||||
2. **Gmail:** Enable 2FA -> generate an App Password. IMAP: `imap.gmail.com:993` (SSL), SMTP: `smtp.gmail.com:587` (STARTTLS).
|
||||
3. **Outlook/Office 365:** IMAP: `outlook.office365.com:993` (SSL), SMTP: `smtp.office365.com:587` (STARTTLS).
|
||||
4. Ensure IMAP access is enabled in your email settings.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
channel_enabled: "email"
|
||||
email_imap_host: "imap.gmail.com"
|
||||
email_imap_port: 993
|
||||
email_imap_username: "bot@gmail.com"
|
||||
email_imap_password: "xxxx-xxxx-xxxx-xxxx"
|
||||
email_imap_mailbox: "INBOX"
|
||||
email_imap_use_ssl: true
|
||||
email_smtp_host: "smtp.gmail.com"
|
||||
email_smtp_port: 587
|
||||
email_smtp_username: "bot@gmail.com"
|
||||
email_smtp_password: "xxxx-xxxx-xxxx-xxxx"
|
||||
email_smtp_use_tls: true
|
||||
email_from_address: "bot@gmail.com"
|
||||
email_poll_interval: 30
|
||||
email_mark_seen: true
|
||||
email_allowed_senders: ""
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `email_imap_host` | `str` | `""` | **Required.** IMAP server address |
|
||||
| `email_imap_port` | `int` | `993` | IMAP port |
|
||||
| `email_imap_username` | `str` | `""` | **Required.** IMAP login username |
|
||||
| `email_imap_password` | `str` | `""` | **Required.** IMAP login password (or app password) |
|
||||
| `email_imap_mailbox` | `str` | `"INBOX"` | Mailbox folder to monitor |
|
||||
| `email_imap_use_ssl` | `bool` | `true` | Use SSL for IMAP connection |
|
||||
| `email_smtp_host` | `str` | `""` | **Required.** SMTP server address |
|
||||
| `email_smtp_port` | `int` | `587` | SMTP port |
|
||||
| `email_smtp_username` | `str` | `""` | **Required.** SMTP login username |
|
||||
| `email_smtp_password` | `str` | `""` | **Required.** SMTP login password |
|
||||
| `email_smtp_use_tls` | `bool` | `true` | Use STARTTLS (`true`) or SSL (`false`) |
|
||||
| `email_from_address` | `str` | `""` | Sender address (defaults to smtp_username) |
|
||||
| `email_poll_interval` | `int` | `30` | IMAP poll interval in seconds |
|
||||
| `email_mark_seen` | `bool` | `true` | Mark emails as read after processing |
|
||||
| `email_max_body_chars` | `int` | `12000` | Max email body chars (truncated beyond) |
|
||||
| `email_subject_prefix` | `str` | `"Re: "` | Reply subject prefix |
|
||||
| `email_allowed_senders` | `str` | `""` | Comma-separated sender email addresses |
|
||||
|
||||
**Env vars:** `EVOSCIENTIST_EMAIL_IMAP_HOST`, `EVOSCIENTIST_EMAIL_IMAP_USERNAME`, `EVOSCIENTIST_EMAIL_IMAP_PASSWORD`, `EVOSCIENTIST_EMAIL_SMTP_HOST`, `EVOSCIENTIST_EMAIL_SMTP_USERNAME`, `EVOSCIENTIST_EMAIL_SMTP_PASSWORD`
|
||||
|
||||
**Technical details:** IMAP polling mode, checks for UNSEEN emails periodically (max 20 per cycle). Supports SSL and STARTTLS. Auto-parses multipart emails (prefers text/plain, falls back text/html -> plain text). Attachments auto-downloaded. Replies set `In-Reply-To` and `References` headers to maintain email threads. Sends HTML + plain text dual format (multipart/alternative), falls back to plain text on HTML failure. IMAP auto-reconnects on disconnect. Auth errors (auth/login/credential) not retried. No public IP needed. Text chunk limit: no limit.
|
||||
|
||||
---
|
||||
|
||||
### iMessage
|
||||
|
||||
**Install:** No extra Python dependencies. Requires the [imsg](https://github.com/anthropics/imsg) CLI tool.
|
||||
|
||||
**Requirements:** macOS only (iMessage is Apple-proprietary). Requires a signed-in Apple ID with iMessage and Full Disk Access permission for the terminal app.
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
1. Install imsg CLI:
|
||||
```bash
|
||||
brew install imsg
|
||||
```
|
||||
2. Verify: `imsg --version`
|
||||
3. Ensure Messages.app is signed in and working on macOS.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
channel_enabled: "imessage"
|
||||
imessage_cli_path: "imsg"
|
||||
imessage_db_path: ""
|
||||
imessage_service: "auto"
|
||||
imessage_region: "US"
|
||||
imessage_allowed_senders: ""
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `imessage_cli_path` | `str` | `"imsg"` | Path to imsg CLI binary |
|
||||
| `imessage_db_path` | `str` | `""` | iMessage database path (empty = default) |
|
||||
| `imessage_service` | `str` | `"auto"` | Send service: `imessage`, `sms`, or `auto` |
|
||||
| `imessage_region` | `str` | `"US"` | Phone number region code |
|
||||
| `imessage_allowed_senders` | `str` | `""` | Comma-separated allowlist (see below) |
|
||||
|
||||
**Allowlist formats:** phone (`+1234567890`), email (`user@example.com`), `chat_id:123`, `chat_guid:iMessage;-;+1234567890`, wildcard `*`.
|
||||
|
||||
**Env vars:** `EVOSCIENTIST_IMESSAGE_CLI_PATH`, `EVOSCIENTIST_IMESSAGE_SERVICE`, `EVOSCIENTIST_IMESSAGE_ALLOWED_SENDERS`
|
||||
|
||||
**Technical details:** JSON-RPC over stdio with imsg CLI. Creates `watch.subscribe` on startup for real-time message streaming (not polling). Supports iMessage + SMS dual channel (`service: auto`). Target resolution supports chat_id, chat_guid, chat_identifier, and phone/email. Attachments read from local paths provided by imsg. Group detection via `is_group` field. RPC errors (AppleScript/permission/not found) not retried; only connection timeouts retried. Plain text format (no Markdown). No public IP needed. Text chunk limit: 4000 chars.
|
||||
|
||||
---
|
||||
|
||||
## Running Multiple Channels
|
||||
|
||||
Comma-separate channel names in the config to enable multiple channels simultaneously:
|
||||
|
||||
```yaml
|
||||
channel_enabled: "telegram,discord,slack"
|
||||
```
|
||||
|
||||
All enabled channels run concurrently via the internal `MessageBus`. Each channel:
|
||||
- Has its own connection lifecycle (connect, reconnect, health check)
|
||||
- Shares the same `InboundConsumer` worker pool and agent instance
|
||||
- Routes outbound replies back to the originating channel automatically
|
||||
|
||||
### Multi-Channel Architecture
|
||||
|
||||
```
|
||||
ChannelManager
|
||||
├── TelegramChannel ──┐
|
||||
├── DiscordChannel ──┤
|
||||
├── SlackChannel ──┤──► MessageBus ──► InboundConsumer ──► Agent
|
||||
├── FeishuChannel ──┤ │
|
||||
└── ... ──┘ ▼
|
||||
OutboundMessage
|
||||
│
|
||||
Dispatcher routes
|
||||
to origin channel
|
||||
```
|
||||
|
||||
### Health Monitoring
|
||||
|
||||
The `ChannelManager` runs a background health check task that monitors all active channels. Access health status via:
|
||||
|
||||
```bash
|
||||
# CLI
|
||||
EvoSci channel status
|
||||
|
||||
# HTTP (if health endpoint is enabled)
|
||||
curl http://localhost:8080/healthz
|
||||
```
|
||||
|
||||
## Docker Deployment
|
||||
|
||||
For webhook-based channels (Feishu, WeChat), Docker simplifies port mapping and process management:
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
COPY . .
|
||||
RUN pip install evoscientist[feishu,wechat]
|
||||
|
||||
# Expose webhook ports
|
||||
EXPOSE 9000 9001
|
||||
|
||||
CMD ["EvoSci", "serve"]
|
||||
```
|
||||
|
||||
```bash
|
||||
docker build -t evoscientist .
|
||||
docker run -d \
|
||||
-p 9000:9000 \
|
||||
-p 9001:9001 \
|
||||
-e EVOSCIENTIST_CHANNEL_ENABLED="feishu,wechat" \
|
||||
-e EVOSCIENTIST_FEISHU_APP_ID="cli_xxx" \
|
||||
-e EVOSCIENTIST_FEISHU_APP_SECRET="xxx" \
|
||||
-e EVOSCIENTIST_WECHAT_BACKEND="wecom" \
|
||||
-e EVOSCIENTIST_WECHAT_WECOM_CORP_ID="ww..." \
|
||||
-e EVOSCIENTIST_WECHAT_WECOM_SECRET="xxx" \
|
||||
evoscientist
|
||||
```
|
||||
|
||||
For polling/WebSocket channels (Telegram, Discord, Slack, DingTalk, QQ), no port mapping is needed:
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
-e EVOSCIENTIST_CHANNEL_ENABLED="telegram" \
|
||||
-e EVOSCIENTIST_TELEGRAM_BOT_TOKEN="123456:ABC-xxx" \
|
||||
evoscientist
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Bot not responding to messages**
|
||||
|
||||
1. Check that `channel_enabled` includes your channel name
|
||||
2. Verify the bot token/credentials are correct: `EvoSci config get telegram_bot_token`
|
||||
3. If using `allowed_senders`, ensure your user ID is listed
|
||||
4. For group chats, ensure the bot is @mentioned (default behavior)
|
||||
5. Check logs for middleware drops: `DedupMiddleware`, `AllowListMiddleware`, or `MentionGatingMiddleware`
|
||||
|
||||
**"channel X not found" or import errors**
|
||||
|
||||
Install the channel-specific dependencies:
|
||||
```bash
|
||||
pip install evoscientist[telegram] # or discord, slack, feishu, etc.
|
||||
```
|
||||
|
||||
**Webhook channels (Feishu, WeChat) not receiving messages**
|
||||
|
||||
1. Ensure the webhook URL is publicly reachable (not behind NAT without port forwarding)
|
||||
2. For local development, use a tunnel: `ngrok http 9000` or [natapp](https://natapp.cn/) for China
|
||||
3. Verify the callback URL matches exactly (including path: `/webhook/event` for Feishu, `/wechat/callback` for WeChat)
|
||||
4. Check that signature verification tokens match between the platform config and your local config
|
||||
|
||||
**WeChat API error `60020`**
|
||||
|
||||
Add your server's public IP to the WeCom app's **Trusted IP** list in the admin console.
|
||||
|
||||
**Token refresh failures**
|
||||
|
||||
- Feishu/WeChat/DingTalk tokens auto-refresh with a 5-minute safety margin before expiry
|
||||
- If the refresh endpoint is unreachable (network issues), messages will fail until the next successful refresh
|
||||
- Check proxy settings if your server requires a proxy to reach external APIs
|
||||
|
||||
**Duplicate responses**
|
||||
|
||||
- The `DedupMiddleware` prevents most duplicates using a 60-second LRU cache
|
||||
- If you see duplicates, check if the platform is sending the same message with different IDs (some platforms retry delivery)
|
||||
|
||||
**Messages truncated**
|
||||
|
||||
- Each platform has a max text length (see Capability Matrix)
|
||||
- Long responses are automatically chunked at paragraph/code-block boundaries
|
||||
- Adjust `text_chunk_limit` in the channel config if needed
|
||||
|
||||
### Debug Logging
|
||||
|
||||
Enable debug logs for the channel subsystem:
|
||||
|
||||
```bash
|
||||
export EVOSCIENTIST_LOG_LEVEL=DEBUG
|
||||
# or
|
||||
EvoSci config set log_level debug
|
||||
```
|
||||
|
||||
Channel-specific log output is prefixed with the module path (e.g., `EvoScientist.channels.telegram.channel`).
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Communication channels for EvoScientist.
|
||||
|
||||
This module provides an extensible interface for different messaging channels
|
||||
(iMessage, Telegram, Discord, Slack, WeChat, DingTalk, Feishu, Email, QQ, Signal) to communicate with the EvoScientist agent.
|
||||
"""
|
||||
|
||||
from .base import Channel, IncomingMessage, OutgoingMessage, RawIncoming, chunk_text
|
||||
from .bus import InboundMessage, MessageBus, OutboundMessage
|
||||
from .capabilities import ChannelCapabilities
|
||||
from .channel_manager import (
|
||||
ChannelManager,
|
||||
available_channels,
|
||||
create_channel,
|
||||
register_channel,
|
||||
)
|
||||
from .consumer import InboundConsumer
|
||||
from .formatter import UnifiedFormatter
|
||||
from .middleware import TypingManager
|
||||
from .plugin import ChannelMeta, ChannelPlugin, ReloadPolicy
|
||||
from .standalone import run_standalone
|
||||
|
||||
# Backward compat: ChannelServer is now Channel itself
|
||||
ChannelServer = Channel
|
||||
|
||||
__all__ = [
|
||||
"Channel",
|
||||
# New modules
|
||||
"ChannelCapabilities",
|
||||
"ChannelManager",
|
||||
"ChannelMeta",
|
||||
# Plugin architecture
|
||||
"ChannelPlugin",
|
||||
"ChannelServer",
|
||||
"InboundConsumer",
|
||||
"InboundMessage",
|
||||
"IncomingMessage",
|
||||
"MessageBus",
|
||||
"OutboundMessage",
|
||||
"OutgoingMessage",
|
||||
"RawIncoming",
|
||||
"ReloadPolicy",
|
||||
"TypingManager",
|
||||
"UnifiedFormatter",
|
||||
"available_channels",
|
||||
"chunk_text",
|
||||
"create_channel",
|
||||
"register_channel",
|
||||
"run_standalone",
|
||||
]
|
||||
@@ -0,0 +1,6 @@
|
||||
"""Message bus for decoupled channel-agent communication."""
|
||||
|
||||
from .events import InboundMessage, OutboundMessage
|
||||
from .message_bus import MessageBus
|
||||
|
||||
__all__ = ["InboundMessage", "MessageBus", "OutboundMessage"]
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Event types for the message bus."""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class InboundMessage:
|
||||
"""Message received from a chat channel.
|
||||
|
||||
Carries enough context for the bus to route and for the agent
|
||||
to build a session: which channel, who sent it, which chat.
|
||||
"""
|
||||
|
||||
channel: str
|
||||
sender_id: str
|
||||
chat_id: str
|
||||
content: str
|
||||
timestamp: datetime = field(default_factory=datetime.now)
|
||||
message_id: str = ""
|
||||
media: list[str] = field(default_factory=list)
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
is_group: bool = False
|
||||
was_mentioned: bool = True
|
||||
|
||||
@property
|
||||
def sender(self) -> str:
|
||||
"""Alias for ``sender_id`` (compatibility with IncomingMessage)."""
|
||||
return self.sender_id
|
||||
|
||||
@property
|
||||
def session_key(self) -> str:
|
||||
"""Unique key for session identification: ``channel:chat_id``."""
|
||||
return f"{self.channel}:{self.chat_id}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class OutboundMessage:
|
||||
"""Message to send to a chat channel."""
|
||||
|
||||
channel: str
|
||||
chat_id: str
|
||||
content: str
|
||||
reply_to: str | None = None
|
||||
media: list[str] = field(default_factory=list)
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def recipient(self) -> str:
|
||||
"""Alias for ``chat_id`` (compatibility with OutgoingMessage)."""
|
||||
return self.chat_id
|
||||
@@ -0,0 +1,114 @@
|
||||
"""Async message bus that decouples chat channels from the agent core.
|
||||
|
||||
Channels push messages to the inbound queue; the agent (or any consumer)
|
||||
reads from inbound, processes, and pushes responses to the outbound queue.
|
||||
A background dispatcher routes outbound messages to the correct channel
|
||||
via subscriber callbacks.
|
||||
|
||||
Deduplication is handled at the Channel level (single dedup point).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from ..debug import TraceMixin, debug_trace_enabled
|
||||
from .events import InboundMessage, OutboundMessage
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
OutboundCallback = Callable[[OutboundMessage], Awaitable[None]]
|
||||
|
||||
|
||||
class MessageBus(TraceMixin):
|
||||
"""Async message bus that decouples chat channels from the agent core."""
|
||||
|
||||
name = "bus"
|
||||
|
||||
def __init__(self):
|
||||
self.inbound: asyncio.Queue[InboundMessage] = asyncio.Queue(maxsize=5000)
|
||||
self.outbound: asyncio.Queue[OutboundMessage] = asyncio.Queue(maxsize=5000)
|
||||
self._outbound_subscribers: dict[str, list[OutboundCallback]] = {}
|
||||
self._running = False
|
||||
self._debug_trace = debug_trace_enabled()
|
||||
self._trace_logger = logger
|
||||
|
||||
# ── inbound (channel → agent) ──
|
||||
|
||||
async def publish_inbound(self, msg: InboundMessage) -> None:
|
||||
"""Publish a message from a channel to the agent."""
|
||||
await self.inbound.put(msg)
|
||||
|
||||
async def consume_inbound(self) -> InboundMessage:
|
||||
"""Consume the next inbound message (blocks until available)."""
|
||||
return await self.inbound.get()
|
||||
|
||||
# ── outbound (agent → channel) ──
|
||||
|
||||
async def publish_outbound(self, msg: OutboundMessage) -> None:
|
||||
"""Publish a response from the agent to channels."""
|
||||
await self.outbound.put(msg)
|
||||
|
||||
async def consume_outbound(self) -> OutboundMessage:
|
||||
"""Consume the next outbound message (blocks until available)."""
|
||||
return await self.outbound.get()
|
||||
|
||||
# ── subscriber routing ──
|
||||
|
||||
def subscribe_outbound(
|
||||
self,
|
||||
channel: str,
|
||||
callback: OutboundCallback,
|
||||
) -> None:
|
||||
"""Register a callback for outbound messages targeting *channel*."""
|
||||
if channel not in self._outbound_subscribers:
|
||||
self._outbound_subscribers[channel] = []
|
||||
self._outbound_subscribers[channel].append(callback)
|
||||
|
||||
async def dispatch_outbound(self) -> None:
|
||||
"""Route outbound messages to subscribed channels.
|
||||
|
||||
Run as a background task — loops until :meth:`stop` is called.
|
||||
"""
|
||||
self._running = True
|
||||
while self._running:
|
||||
try:
|
||||
msg = await asyncio.wait_for(
|
||||
self.outbound.get(),
|
||||
timeout=1.0,
|
||||
)
|
||||
except TimeoutError:
|
||||
continue
|
||||
subscribers = self._outbound_subscribers.get(msg.channel, [])
|
||||
if not subscribers:
|
||||
self._trace_event(
|
||||
"bus_dispatch_drop",
|
||||
target_channel=msg.channel,
|
||||
reason="no_subscriber",
|
||||
chat_id=msg.chat_id,
|
||||
)
|
||||
logger.warning(f"No subscriber for channel: {msg.channel}")
|
||||
continue
|
||||
for callback in subscribers:
|
||||
try:
|
||||
await callback(msg)
|
||||
except Exception as e:
|
||||
self._trace_event(
|
||||
"bus_dispatch_error",
|
||||
target_channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
error_type=type(e).__name__,
|
||||
)
|
||||
logger.error(f"Error dispatching to {msg.channel}: {e}")
|
||||
|
||||
def stop(self) -> None:
|
||||
"""Stop the dispatcher loop."""
|
||||
self._running = False
|
||||
|
||||
@property
|
||||
def inbound_size(self) -> int:
|
||||
return self.inbound.qsize()
|
||||
|
||||
@property
|
||||
def outbound_size(self) -> int:
|
||||
return self.outbound.qsize()
|
||||
@@ -0,0 +1,220 @@
|
||||
"""Channel capabilities declaration system.
|
||||
|
||||
Each channel declares its capabilities via a ChannelCapabilities dataclass,
|
||||
enabling the framework to adapt behavior automatically (formatting, reactions,
|
||||
streaming, threading, etc.) without per-channel branching in core logic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
|
||||
FormatType = Literal["html", "markdown", "slack_mrkdwn", "discord", "plain"]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ChannelCapabilities:
|
||||
"""Immutable declaration of what a channel supports.
|
||||
|
||||
Set once as a class attribute on each Channel subclass.
|
||||
The framework inspects these at runtime to auto-configure behavior.
|
||||
"""
|
||||
|
||||
# ── Messaging features ──────────────────────────────────────────
|
||||
format_type: FormatType = "plain"
|
||||
max_text_length: int = 4096
|
||||
max_file_size: int = 20 * 1024 * 1024 # 20 MB
|
||||
|
||||
# ── Interaction capabilities ────────────────────────────────────
|
||||
streaming: bool = False # edit-in-place streaming output
|
||||
threading: bool = False # message threads / topics
|
||||
reactions: bool = False # emoji reactions on messages
|
||||
typing: bool = False # typing indicator API
|
||||
inline_buttons: bool = False # inline keyboard / action buttons
|
||||
|
||||
# ── Media capabilities ──────────────────────────────────────────
|
||||
media_send: bool = False # can send files/images
|
||||
media_receive: bool = False # can receive files/images
|
||||
voice: bool = False # platform has voice/audio messages that arrive as downloadable files (receive only, not bot sending)
|
||||
stickers: bool = False # supports sticker receive (not bot sending)
|
||||
location: bool = False # supports location message receive (not bot sending)
|
||||
video: bool = False # video messages
|
||||
|
||||
# ── Group features ──────────────────────────────────────────────
|
||||
groups: bool = False # group chat support
|
||||
mentions: bool = False # @mention detection
|
||||
|
||||
# ── Rich text ───────────────────────────────────────────────────
|
||||
markdown: bool = False # supports Markdown rendering
|
||||
html: bool = False # supports HTML rendering
|
||||
|
||||
# ── Extended capabilities ────────────────────────────────────────
|
||||
chat_types: tuple[str, ...] = () # ("direct", "group", "channel", "thread")
|
||||
edit: bool = False # message editing after send
|
||||
unsend: bool = False # message recall / unsend
|
||||
block_streaming: bool = False # block edit-in-place streaming
|
||||
native_commands: bool = False # platform-native slash commands
|
||||
polls: bool = False # poll / vote messages
|
||||
|
||||
def supports(self, feature: str) -> bool:
|
||||
"""Check if a feature is supported by name."""
|
||||
return getattr(self, feature, False)
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Pre-built capability profiles for each channel
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
TELEGRAM = ChannelCapabilities(
|
||||
format_type="html",
|
||||
max_text_length=4000,
|
||||
streaming=False, # could edit messages, but not implemented yet
|
||||
threading=False, # topics exist but not used yet
|
||||
reactions=True,
|
||||
typing=True,
|
||||
media_send=True,
|
||||
media_receive=True,
|
||||
voice=True,
|
||||
stickers=True,
|
||||
location=True,
|
||||
groups=True,
|
||||
mentions=True,
|
||||
html=True,
|
||||
chat_types=("direct", "group", "channel"),
|
||||
edit=True,
|
||||
unsend=True,
|
||||
native_commands=True,
|
||||
polls=True,
|
||||
)
|
||||
|
||||
DISCORD = ChannelCapabilities(
|
||||
format_type="discord",
|
||||
max_text_length=2000,
|
||||
streaming=False,
|
||||
threading=True,
|
||||
reactions=True,
|
||||
typing=True,
|
||||
media_send=True,
|
||||
media_receive=True,
|
||||
voice=False, # no distinct voice message type in Discord bot API
|
||||
groups=True,
|
||||
mentions=True,
|
||||
markdown=True,
|
||||
chat_types=("direct", "group", "thread"),
|
||||
edit=True,
|
||||
unsend=True,
|
||||
native_commands=True,
|
||||
polls=True,
|
||||
)
|
||||
|
||||
SLACK = ChannelCapabilities(
|
||||
format_type="slack_mrkdwn",
|
||||
max_text_length=4000,
|
||||
streaming=False,
|
||||
threading=True,
|
||||
reactions=True,
|
||||
typing=False, # no native typing API; workaround via post+delete "..." message
|
||||
media_send=True,
|
||||
media_receive=True,
|
||||
voice=False, # no distinct voice message type in Slack bot API
|
||||
groups=True,
|
||||
mentions=True,
|
||||
chat_types=("direct", "group", "thread"),
|
||||
edit=True,
|
||||
unsend=True,
|
||||
native_commands=True,
|
||||
)
|
||||
|
||||
FEISHU = ChannelCapabilities(
|
||||
format_type="markdown",
|
||||
max_text_length=4096,
|
||||
reactions=True,
|
||||
typing=False, # no typing API
|
||||
media_send=True,
|
||||
media_receive=True,
|
||||
voice=True,
|
||||
stickers=True,
|
||||
groups=True,
|
||||
mentions=True,
|
||||
markdown=True,
|
||||
chat_types=("direct", "group"),
|
||||
edit=True,
|
||||
unsend=True,
|
||||
)
|
||||
|
||||
DINGTALK = ChannelCapabilities(
|
||||
format_type="markdown",
|
||||
max_text_length=4096,
|
||||
typing=False, # no typing API for bots
|
||||
media_send=True,
|
||||
media_receive=True,
|
||||
voice=True,
|
||||
groups=True,
|
||||
mentions=True,
|
||||
markdown=True,
|
||||
chat_types=("direct", "group"),
|
||||
)
|
||||
|
||||
QQ = ChannelCapabilities(
|
||||
format_type="plain",
|
||||
max_text_length=4096,
|
||||
typing=False, # no typing API for QQ bots
|
||||
media_send=True,
|
||||
media_receive=True,
|
||||
voice=False, # qq-botpy does not expose voice as a distinct message type
|
||||
groups=True,
|
||||
mentions=True,
|
||||
chat_types=("direct", "group", "channel"),
|
||||
unsend=True,
|
||||
)
|
||||
|
||||
WECHAT = ChannelCapabilities(
|
||||
format_type="markdown", # WeCom supports markdown
|
||||
max_text_length=4096,
|
||||
typing=False, # no typing API
|
||||
media_send=True,
|
||||
media_receive=True,
|
||||
voice=True,
|
||||
location=True,
|
||||
groups=True,
|
||||
mentions=True,
|
||||
markdown=True,
|
||||
chat_types=("direct", "group"),
|
||||
unsend=True,
|
||||
)
|
||||
|
||||
SIGNAL = ChannelCapabilities(
|
||||
format_type="plain",
|
||||
max_text_length=4096,
|
||||
reactions=True,
|
||||
typing=True,
|
||||
media_send=True,
|
||||
media_receive=True,
|
||||
voice=True,
|
||||
groups=True,
|
||||
mentions=True,
|
||||
chat_types=("direct", "group"),
|
||||
)
|
||||
|
||||
EMAIL = ChannelCapabilities(
|
||||
format_type="html",
|
||||
max_text_length=999_999, # no practical limit
|
||||
media_send=True,
|
||||
media_receive=True,
|
||||
html=True,
|
||||
chat_types=("direct",),
|
||||
)
|
||||
|
||||
IMESSAGE = ChannelCapabilities(
|
||||
format_type="plain",
|
||||
max_text_length=999_999,
|
||||
typing=False, # Apple does not expose typing indicator API
|
||||
media_send=True,
|
||||
media_receive=True,
|
||||
voice=True,
|
||||
groups=True,
|
||||
mentions=False, # iMessage has no @mention concept
|
||||
reactions=False, # imsg CLI cannot send tapback reactions
|
||||
chat_types=("direct", "group"),
|
||||
)
|
||||
@@ -0,0 +1,128 @@
|
||||
"""Base configuration for all channel implementations.
|
||||
|
||||
Provides common fields shared across channels, reducing duplication.
|
||||
Channel-specific configs inherit from BaseChannelConfig.
|
||||
|
||||
Also provides ready-made ConfigAdapter implementations for the two most
|
||||
common account patterns:
|
||||
|
||||
- ``SingleAccountConfigAdapter`` — one account per channel (default).
|
||||
- ``MultiAccountConfigAdapter`` — multiple accounts from a config dict.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class BaseChannelConfig:
|
||||
"""Common configuration fields for all channels.
|
||||
|
||||
Subclass this for channel-specific configs. Only add fields
|
||||
here that are used by 3+ channels.
|
||||
"""
|
||||
|
||||
allowed_senders: set[str] | None = None
|
||||
allowed_channels: set[str] | None = None
|
||||
text_chunk_limit: int = 4096
|
||||
proxy: str | None = None
|
||||
include_attachments: bool = True
|
||||
debug_trace: bool = False
|
||||
accounts: dict | None = None # multi-account config mapping
|
||||
|
||||
|
||||
class SingleAccountConfigAdapter:
|
||||
"""For channels that only ever have one account (most channels).
|
||||
|
||||
Returns a single ``"default"`` account whose config is the entire
|
||||
channel config object. This is the zero-change default: existing
|
||||
single-account channels get multi-account support for free.
|
||||
"""
|
||||
|
||||
def list_account_ids(self, config: Any) -> list[str]:
|
||||
return ["default"]
|
||||
|
||||
def resolve_account(
|
||||
self,
|
||||
config: Any,
|
||||
account_id: str | None = None,
|
||||
) -> Any:
|
||||
return config
|
||||
|
||||
def is_enabled(self, account: Any, config: Any) -> bool:
|
||||
return True
|
||||
|
||||
def is_configured(self, account: Any, config: Any) -> bool:
|
||||
"""Check that the account has at least some non-None values."""
|
||||
if account is None:
|
||||
return False
|
||||
if isinstance(account, dict):
|
||||
return bool(account)
|
||||
# dataclass / object — check that at least one field is truthy
|
||||
if hasattr(account, "__dataclass_fields__"):
|
||||
return any(getattr(account, f, None) for f in account.__dataclass_fields__)
|
||||
return True
|
||||
|
||||
|
||||
class MultiAccountConfigAdapter:
|
||||
"""For channels that support multiple accounts.
|
||||
|
||||
Expects the channel config to contain a mapping of accounts under
|
||||
a configurable key (default ``"accounts"``). Each entry is keyed
|
||||
by account id and holds account-specific settings.
|
||||
|
||||
Example config structure::
|
||||
|
||||
{
|
||||
"accounts": {
|
||||
"bot1": {"token": "...", "enabled": true},
|
||||
"bot2": {"token": "...", "enabled": false},
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
accounts_key: str = "accounts",
|
||||
required_fields: list[str] | None = None,
|
||||
) -> None:
|
||||
self._accounts_key = accounts_key
|
||||
self._required_fields = required_fields or []
|
||||
|
||||
def _get_accounts_map(self, config: Any) -> dict[str, Any]:
|
||||
"""Extract the accounts mapping from config."""
|
||||
if isinstance(config, dict):
|
||||
return config.get(self._accounts_key, {})
|
||||
return getattr(config, self._accounts_key, None) or {}
|
||||
|
||||
def list_account_ids(self, config: Any) -> list[str]:
|
||||
return list(self._get_accounts_map(config).keys())
|
||||
|
||||
def resolve_account(
|
||||
self,
|
||||
config: Any,
|
||||
account_id: str | None = None,
|
||||
) -> Any:
|
||||
accounts = self._get_accounts_map(config)
|
||||
if account_id is None:
|
||||
# Return the first account, or empty dict
|
||||
return next(iter(accounts.values()), {})
|
||||
return accounts.get(account_id, {})
|
||||
|
||||
def is_enabled(self, account: Any, config: Any) -> bool:
|
||||
if isinstance(account, dict):
|
||||
return account.get("enabled", True)
|
||||
return getattr(account, "enabled", True)
|
||||
|
||||
def is_configured(self, account: Any, config: Any) -> bool:
|
||||
if not account:
|
||||
return False
|
||||
for f in self._required_fields:
|
||||
if isinstance(account, dict):
|
||||
if not account.get(f):
|
||||
return False
|
||||
elif not getattr(account, f, None):
|
||||
return False
|
||||
return True
|
||||
@@ -0,0 +1,888 @@
|
||||
"""Unified inbound message consumer.
|
||||
|
||||
Provides :class:`InboundConsumer` — a single class that consumes
|
||||
inbound messages from the :class:`MessageBus`, runs them through
|
||||
the agent, and publishes outbound responses. This replaces the
|
||||
inline consumer loops that were duplicated in ``cli.py`` and
|
||||
``standalone.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
import uuid
|
||||
from collections import OrderedDict
|
||||
from collections.abc import AsyncIterator, Callable
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, TypeVar
|
||||
|
||||
from .base import Channel
|
||||
from .bus import MessageBus
|
||||
from .bus.events import InboundMessage, OutboundMessage
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
_MAX_CHAT_LOCKS = 10_000
|
||||
_MAX_SESSIONS = 10_000
|
||||
_MAX_HITL_ROUNDS = 50
|
||||
_HITL_APPROVAL_TIMEOUT = 120.0 # seconds to wait for HITL approval reply
|
||||
_ASK_USER_TIMEOUT = (
|
||||
300.0 # seconds to wait for ask_user reply (longer for thinking time)
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ConsumerMetrics:
|
||||
"""Cumulative processing counters for the consumer."""
|
||||
|
||||
total_processed: int = 0
|
||||
total_successes: int = 0
|
||||
total_failures: int = 0
|
||||
total_timeouts: int = 0
|
||||
|
||||
|
||||
async def _timeout_aiter(
|
||||
agen: AsyncIterator[T],
|
||||
idle_timeout: float,
|
||||
) -> AsyncIterator[T]:
|
||||
"""Wrap an async iterator with a per-yield idle timeout.
|
||||
|
||||
If ``__anext__()`` does not produce a value within *idle_timeout*
|
||||
seconds, :class:`asyncio.TimeoutError` is raised. Continuous
|
||||
yielding resets the timer each time, so only a truly stalled
|
||||
generator will trigger the timeout.
|
||||
"""
|
||||
ait = agen.__aiter__()
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
item = await asyncio.wait_for(ait.__anext__(), timeout=idle_timeout)
|
||||
except StopAsyncIteration:
|
||||
return
|
||||
yield item
|
||||
finally:
|
||||
if hasattr(ait, "aclose"):
|
||||
await ait.aclose()
|
||||
|
||||
|
||||
def _format_todo_list(todos: list[dict]) -> str:
|
||||
"""Format todo items as a numbered list."""
|
||||
lines = ["\U0001f4cb Todo List\n"] # 📋
|
||||
for i, item in enumerate(todos, 1):
|
||||
content = item.get("content", "")
|
||||
lines.append(f"{i}. {content}")
|
||||
lines.append(f"\n\U0001f680 {len(todos)} tasks") # 🚀
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _join_subagent_text(buffers: dict[str, tuple[str, list[str]]]) -> str:
|
||||
"""Join sub-agent text buffers into a single fallback string.
|
||||
|
||||
*buffers* maps ``instance_id`` → ``(display_name, chunks)``.
|
||||
|
||||
When only one instance produced text, return its content directly.
|
||||
When multiple instances share the same display name, number them
|
||||
(e.g. ``[research-agent #1]``, ``[research-agent #2]``).
|
||||
"""
|
||||
if not buffers:
|
||||
return ""
|
||||
if len(buffers) == 1:
|
||||
_display_name, chunks = next(iter(buffers.values()))
|
||||
return "".join(chunks)
|
||||
|
||||
# Group by display_name to detect same-name instances
|
||||
name_groups: dict[str, list[list[str]]] = {}
|
||||
for _instance_id, (display_name, chunks) in buffers.items():
|
||||
name_groups.setdefault(display_name, []).append(chunks)
|
||||
|
||||
sections: list[str] = []
|
||||
for display_name, chunk_lists in name_groups.items():
|
||||
if len(chunk_lists) == 1:
|
||||
sections.append(f"[{display_name}]: {''.join(chunk_lists[0])}")
|
||||
else:
|
||||
for i, chs in enumerate(chunk_lists, 1):
|
||||
sections.append(f"[{display_name} #{i}]: {''.join(chs)}")
|
||||
return "\n\n".join(sections)
|
||||
|
||||
|
||||
def _should_auto_approve(action_requests: list[dict]) -> bool:
|
||||
"""Check if all action requests can be auto-approved via config.
|
||||
|
||||
Returns True if no manual approval is needed (config auto_approve,
|
||||
non-execute tools, or shell_allow_list match).
|
||||
"""
|
||||
if not action_requests:
|
||||
return True
|
||||
|
||||
try:
|
||||
from ..config.settings import load_config
|
||||
|
||||
cfg = load_config()
|
||||
except Exception:
|
||||
return False # fail-closed
|
||||
|
||||
if cfg.auto_approve:
|
||||
return True
|
||||
|
||||
shell_allow_list = (
|
||||
[s.strip() for s in cfg.shell_allow_list.split(",") if s.strip()]
|
||||
if cfg.shell_allow_list
|
||||
else []
|
||||
)
|
||||
|
||||
for req in action_requests:
|
||||
name = (
|
||||
req.get("name", "") if isinstance(req, dict) else getattr(req, "name", "")
|
||||
)
|
||||
if name != "execute":
|
||||
continue
|
||||
args = (
|
||||
req.get("args", {}) if isinstance(req, dict) else getattr(req, "args", {})
|
||||
)
|
||||
command = args.get("command", "") if isinstance(args, dict) else ""
|
||||
cmd = command.strip()
|
||||
if not any(cmd.startswith(prefix) for prefix in shell_allow_list):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _format_approval_prompt(action_requests: list[dict]) -> str:
|
||||
"""Format an approval prompt as a text message for channel users."""
|
||||
lines = ["\u26a0\ufe0f Approval Required\n"]
|
||||
for i, req in enumerate(action_requests, 1):
|
||||
name = (
|
||||
req.get("name", "") if isinstance(req, dict) else getattr(req, "name", "")
|
||||
)
|
||||
args = (
|
||||
req.get("args", {}) if isinstance(req, dict) else getattr(req, "args", {})
|
||||
)
|
||||
if isinstance(args, dict):
|
||||
command = args.get("command", args.get("path", ""))
|
||||
else:
|
||||
command = ""
|
||||
if command:
|
||||
lines.append(f" {i}. {name}: {command}")
|
||||
else:
|
||||
lines.append(f" {i}. {name}")
|
||||
lines.append("")
|
||||
lines.append("Reply: 1=Approve, 2=Reject, 3=Approve all")
|
||||
lines.append("(Auto-reject in 2 min if no reply)")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _parse_approval_reply(text: str) -> str | None:
|
||||
"""Parse a channel user's reply as an approval decision.
|
||||
|
||||
Returns "approve", "reject", "auto", or None if not recognized.
|
||||
"""
|
||||
t = text.strip().lower()
|
||||
if t in ("1", "y", "yes", "approve", "ok"):
|
||||
return "approve"
|
||||
if t in ("2", "n", "no", "reject"):
|
||||
return "reject"
|
||||
if t in ("3", "a", "auto", "approve all"):
|
||||
return "auto"
|
||||
return None
|
||||
|
||||
|
||||
@dataclass
|
||||
class _PendingInterrupt:
|
||||
"""Stored state for a pending HITL interrupt awaiting channel user reply."""
|
||||
|
||||
thread_id: str
|
||||
action_requests: list
|
||||
event: asyncio.Event # set when user replies
|
||||
decision: str | None = None # "approve", "reject", "auto"
|
||||
|
||||
|
||||
@dataclass
|
||||
class _PendingAskUserReply:
|
||||
"""Stored state for a pending ask_user question awaiting channel user reply."""
|
||||
|
||||
event: asyncio.Event # set when user replies
|
||||
reply: str | None = None # raw reply text
|
||||
|
||||
|
||||
class InboundConsumer:
|
||||
"""Consume inbound messages from the bus, process via agent, publish outbound.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
bus:
|
||||
The MessageBus to consume from / publish to.
|
||||
manager:
|
||||
The ChannelManager (used to look up channel instances).
|
||||
agent:
|
||||
The agent object (must support ``stream_agent_events``).
|
||||
thread_id:
|
||||
Default thread ID for agent conversations.
|
||||
send_thinking:
|
||||
Whether to forward thinking messages to the channel.
|
||||
on_message_received:
|
||||
Optional callback ``(msg: InboundMessage) -> None`` invoked when
|
||||
a message is consumed (e.g. for CLI Rich display).
|
||||
on_streaming_event:
|
||||
Optional callback ``(event: dict) -> None`` invoked for each
|
||||
streaming event from the agent.
|
||||
on_message_sent:
|
||||
Optional callback ``(msg: OutboundMessage) -> None`` invoked when
|
||||
the outbound message is published.
|
||||
inference_timeout:
|
||||
Per-yield idle timeout in seconds for the agent stream. If the
|
||||
agent produces no event for this long, the inference is aborted.
|
||||
max_concurrent:
|
||||
Number of worker coroutines (= max parallel inferences).
|
||||
max_pending:
|
||||
Maximum depth of the internal work queue. When full, the
|
||||
consumer loop blocks (back-pressure).
|
||||
drain_timeout:
|
||||
Seconds to wait for in-flight workers to finish during ``stop()``.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
bus: MessageBus,
|
||||
manager: Any,
|
||||
agent: Any,
|
||||
thread_id: str,
|
||||
*,
|
||||
send_thinking: bool = False,
|
||||
on_message_received: Callable[[InboundMessage], None] | None = None,
|
||||
on_streaming_event: Callable[[dict], None] | None = None,
|
||||
on_message_sent: Callable[[OutboundMessage], None] | None = None,
|
||||
inference_timeout: float = 300.0,
|
||||
max_concurrent: int = 5,
|
||||
max_pending: int = 50,
|
||||
drain_timeout: float = 30.0,
|
||||
):
|
||||
self.bus = bus
|
||||
self.manager = manager
|
||||
self.agent = agent
|
||||
self.thread_id = thread_id
|
||||
self.send_thinking = send_thinking
|
||||
self._on_message_received = on_message_received
|
||||
self._on_streaming_event = on_streaming_event
|
||||
self._on_message_sent = on_message_sent
|
||||
self._sessions: OrderedDict[str, str] = (
|
||||
OrderedDict()
|
||||
) # sender_id -> thread_id (LRU)
|
||||
|
||||
# Per-chat locks: same chat is processed serially (bounded)
|
||||
self._chat_locks: dict[str, asyncio.Lock] = {}
|
||||
|
||||
# Inference timeout
|
||||
self._inference_timeout = inference_timeout
|
||||
|
||||
# Worker pool
|
||||
self._max_concurrent = max_concurrent
|
||||
self._work_queue: asyncio.Queue[InboundMessage | None] = asyncio.Queue(
|
||||
maxsize=max_pending,
|
||||
)
|
||||
self._workers: list[asyncio.Task] = []
|
||||
self._stopping = False
|
||||
self._drain_timeout = drain_timeout
|
||||
|
||||
# Metrics
|
||||
self._metrics = ConsumerMetrics()
|
||||
|
||||
# HITL: pending interrupts per session_key, and auto-approve sessions
|
||||
self._pending_interrupts: dict[str, _PendingInterrupt] = {}
|
||||
self._auto_approve_sessions: set[str] = set()
|
||||
|
||||
# ask_user: pending reply per session_key
|
||||
self._pending_ask_user_replies: dict[str, _PendingAskUserReply] = {}
|
||||
|
||||
def _get_thread_id(self, sender_id: str) -> str:
|
||||
"""Get or create a thread ID for the given sender.
|
||||
|
||||
Uses LRU ordering: recently accessed senders are moved to the
|
||||
end, so eviction always removes the least-recently-active sender.
|
||||
"""
|
||||
if sender_id in self._sessions:
|
||||
self._sessions.move_to_end(sender_id)
|
||||
return self._sessions[sender_id]
|
||||
|
||||
if len(self._sessions) >= _MAX_SESSIONS:
|
||||
# Evict the least-recently-used entry
|
||||
self._sessions.popitem(last=False)
|
||||
if self.thread_id:
|
||||
self._sessions[sender_id] = f"{self.thread_id}:{sender_id}"
|
||||
else:
|
||||
self._sessions[sender_id] = str(uuid.uuid4())
|
||||
return self._sessions[sender_id]
|
||||
|
||||
def _get_channel(self, channel_name: str) -> Channel | None:
|
||||
"""Look up the channel by name from the manager."""
|
||||
return self.manager.get_channel(channel_name)
|
||||
|
||||
# ── lifecycle ──
|
||||
|
||||
async def run(self) -> None:
|
||||
"""Main consumer loop — runs until ``stop()`` or cancellation.
|
||||
|
||||
Spawns *max_concurrent* worker coroutines that pull from an
|
||||
internal bounded queue. The loop reads from the bus and feeds
|
||||
the queue; when the queue is full the loop blocks (back-pressure).
|
||||
"""
|
||||
self._stopping = False
|
||||
self._workers = [
|
||||
asyncio.create_task(self._worker(i)) for i in range(self._max_concurrent)
|
||||
]
|
||||
try:
|
||||
while not self._stopping:
|
||||
try:
|
||||
msg = await asyncio.wait_for(
|
||||
self.bus.consume_inbound(),
|
||||
timeout=1.0,
|
||||
)
|
||||
except TimeoutError:
|
||||
continue
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
if self._stopping:
|
||||
break
|
||||
await self._work_queue.put(msg) # blocks when full (back-pressure)
|
||||
finally:
|
||||
if not self._stopping:
|
||||
await self.stop()
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Gracefully drain in-flight work and shut down workers."""
|
||||
self._stopping = True
|
||||
logger.info("Consumer stopping: draining in-flight messages...")
|
||||
pending_count = self._work_queue.qsize()
|
||||
|
||||
# Send a None sentinel per worker so each exits its loop
|
||||
for _ in self._workers:
|
||||
try:
|
||||
self._work_queue.put_nowait(None)
|
||||
except asyncio.QueueFull:
|
||||
pass
|
||||
|
||||
# Wait for workers to finish, then force-cancel stragglers
|
||||
if self._workers:
|
||||
done, still_running = await asyncio.wait(
|
||||
self._workers,
|
||||
timeout=self._drain_timeout,
|
||||
)
|
||||
for task in still_running:
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except (asyncio.CancelledError, Exception):
|
||||
pass
|
||||
logger.info(
|
||||
f"Consumer drain: {len(done)} finished, "
|
||||
f"{len(still_running)} force-cancelled, "
|
||||
f"{pending_count} were pending"
|
||||
)
|
||||
self._workers.clear()
|
||||
|
||||
# ── workers ──
|
||||
|
||||
async def _worker(self, worker_id: int) -> None:
|
||||
"""Pull messages from the work queue and process them."""
|
||||
while True:
|
||||
msg = await self._work_queue.get()
|
||||
if msg is None:
|
||||
break # shutdown sentinel
|
||||
try:
|
||||
await self._handle_message(msg)
|
||||
except Exception:
|
||||
logger.exception(f"Worker {worker_id} unhandled error")
|
||||
finally:
|
||||
self._work_queue.task_done()
|
||||
|
||||
async def _handle_message(self, msg: InboundMessage) -> None:
|
||||
"""Process a single inbound message."""
|
||||
if self._on_message_received:
|
||||
try:
|
||||
self._on_message_received(msg)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
channel = self._get_channel(msg.channel)
|
||||
thread_id = self._get_thread_id(msg.sender_id)
|
||||
session_key = msg.session_key # "channel:chat_id"
|
||||
|
||||
# Lazily create per-chat lock; evict stale locks when too many
|
||||
if session_key not in self._chat_locks:
|
||||
self._chat_locks[session_key] = asyncio.Lock()
|
||||
if len(self._chat_locks) > _MAX_CHAT_LOCKS:
|
||||
self._evict_chat_locks()
|
||||
|
||||
self._metrics.total_processed += 1
|
||||
|
||||
# ask_user: check if this message is a reply to a pending question.
|
||||
# Must be checked BEFORE HITL approval — any text is a valid answer.
|
||||
if session_key in self._pending_ask_user_replies:
|
||||
pending_ask = self._pending_ask_user_replies[session_key]
|
||||
pending_ask.reply = msg.content
|
||||
pending_ask.event.set()
|
||||
return # consumed as ask_user answer
|
||||
|
||||
# HITL: check if this message is a reply to a pending approval
|
||||
if session_key in self._pending_interrupts:
|
||||
pending = self._pending_interrupts[session_key]
|
||||
decision = _parse_approval_reply(msg.content)
|
||||
if decision is not None:
|
||||
pending.decision = decision
|
||||
pending.event.set()
|
||||
return # don't process as a new agent message
|
||||
# Unrecognized reply — treat as new message, cancel pending
|
||||
pending.decision = "reject"
|
||||
pending.event.set()
|
||||
del self._pending_interrupts[session_key]
|
||||
|
||||
async with self._chat_locks[session_key]:
|
||||
await self._stream_with_hitl(msg, channel, thread_id, session_key)
|
||||
|
||||
async def _stream_with_hitl(
|
||||
self,
|
||||
msg: InboundMessage,
|
||||
channel: Channel | None,
|
||||
thread_id: str,
|
||||
session_key: str,
|
||||
) -> None:
|
||||
"""Stream agent events with HITL interrupt handling."""
|
||||
from ..stream.events import stream_agent_events
|
||||
|
||||
stream_input: Any = msg.content
|
||||
_t0 = time.monotonic()
|
||||
|
||||
try:
|
||||
if channel:
|
||||
await channel.start_typing(msg.chat_id)
|
||||
|
||||
for _hitl_round in range(_MAX_HITL_ROUNDS):
|
||||
final_content = ""
|
||||
thinking_buffer: list[str] = []
|
||||
todo_sent = False
|
||||
subagent_text_buffers: dict[str, tuple[str, list[str]]] = {}
|
||||
thinking_sent = False
|
||||
interrupt_data: dict | None = None
|
||||
|
||||
async for event in _timeout_aiter(
|
||||
stream_agent_events(
|
||||
self.agent,
|
||||
stream_input,
|
||||
thread_id,
|
||||
media=msg.media or None
|
||||
if isinstance(stream_input, str)
|
||||
else None,
|
||||
),
|
||||
self._inference_timeout,
|
||||
):
|
||||
event_type = event.get("type")
|
||||
|
||||
if self._on_streaming_event:
|
||||
try:
|
||||
self._on_streaming_event(event)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if event_type == "thinking":
|
||||
thinking_text = event.get("content", "")
|
||||
if thinking_text:
|
||||
thinking_buffer.append(thinking_text)
|
||||
|
||||
elif event_type == "tool_call":
|
||||
if event.get("name") == "write_todos" and not todo_sent:
|
||||
todos = event.get("args", {}).get("todos", [])
|
||||
if todos and channel:
|
||||
if thinking_buffer and not thinking_sent:
|
||||
full_thinking = "".join(thinking_buffer)
|
||||
if full_thinking:
|
||||
await channel.send_thinking_message(
|
||||
msg.sender_id,
|
||||
full_thinking,
|
||||
msg.metadata,
|
||||
)
|
||||
thinking_sent = True
|
||||
thinking_buffer.clear()
|
||||
await channel.send_todo_message(
|
||||
msg.sender_id,
|
||||
_format_todo_list(todos),
|
||||
msg.metadata,
|
||||
)
|
||||
todo_sent = True
|
||||
|
||||
elif event_type == "text":
|
||||
final_content += event.get("content", "")
|
||||
|
||||
elif event_type == "progress":
|
||||
# Internal agent planning — ignore for channel delivery.
|
||||
# This text should NOT appear in the user-facing response.
|
||||
pass
|
||||
|
||||
elif event_type == "subagent_text":
|
||||
sa_name = event.get("subagent", "unknown")
|
||||
instance_id = event.get("instance_id") or sa_name
|
||||
if instance_id not in subagent_text_buffers:
|
||||
subagent_text_buffers[instance_id] = (sa_name, [])
|
||||
subagent_text_buffers[instance_id][1].append(
|
||||
event.get("content", "")
|
||||
)
|
||||
|
||||
elif event_type == "done":
|
||||
final_content = event.get("content", "") or final_content
|
||||
|
||||
elif event_type == "interrupt":
|
||||
interrupt_data = event
|
||||
break # exit async for to handle interrupt
|
||||
|
||||
elif event_type == "ask_user":
|
||||
interrupt_data = event
|
||||
break # exit async for to handle ask_user
|
||||
|
||||
# Flush thinking
|
||||
if thinking_buffer and not thinking_sent and channel:
|
||||
full_thinking = "".join(thinking_buffer)
|
||||
if full_thinking:
|
||||
await channel.send_thinking_message(
|
||||
msg.sender_id,
|
||||
full_thinking,
|
||||
msg.metadata,
|
||||
)
|
||||
|
||||
# No interrupt — normal completion
|
||||
if interrupt_data is None:
|
||||
outbound = OutboundMessage(
|
||||
channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
content=final_content
|
||||
or _join_subagent_text(subagent_text_buffers)
|
||||
or "No response",
|
||||
reply_to=msg.message_id or None,
|
||||
metadata=msg.metadata,
|
||||
)
|
||||
await self.bus.publish_outbound(outbound)
|
||||
self._metrics.total_successes += 1
|
||||
elapsed = time.monotonic() - _t0
|
||||
logger.info(
|
||||
"stream completed: %d chars, %.2fs, session=%s",
|
||||
len(outbound.content),
|
||||
elapsed,
|
||||
session_key,
|
||||
)
|
||||
if self._on_message_sent:
|
||||
try:
|
||||
self._on_message_sent(outbound)
|
||||
except Exception:
|
||||
pass
|
||||
return # done
|
||||
|
||||
# ask_user: send questions to channel user, collect answers
|
||||
if interrupt_data.get("type") == "ask_user":
|
||||
result = await self._resolve_ask_user(
|
||||
msg,
|
||||
interrupt_data,
|
||||
session_key,
|
||||
)
|
||||
from langgraph.types import Command # type: ignore[import-untyped]
|
||||
|
||||
stream_input = Command(resume=result)
|
||||
continue
|
||||
|
||||
# HITL: resolve the interrupt
|
||||
action_reqs = interrupt_data.get("action_requests", [])
|
||||
n = len(action_reqs) or 1
|
||||
|
||||
# Session auto-approve (user previously chose "Approve all")
|
||||
if session_key in self._auto_approve_sessions:
|
||||
from langgraph.types import Command # type: ignore[import-untyped]
|
||||
|
||||
stream_input = Command(
|
||||
resume={"decisions": [{"type": "approve"} for _ in range(n)]}
|
||||
)
|
||||
continue
|
||||
|
||||
# Config auto-approve (auto_approve, non-execute, allow_list)
|
||||
if _should_auto_approve(action_reqs):
|
||||
from langgraph.types import Command # type: ignore[import-untyped]
|
||||
|
||||
stream_input = Command(
|
||||
resume={"decisions": [{"type": "approve"} for _ in range(n)]}
|
||||
)
|
||||
continue
|
||||
|
||||
# Needs user approval — send prompt to channel
|
||||
prompt_text = _format_approval_prompt(action_reqs)
|
||||
await self.bus.publish_outbound(
|
||||
OutboundMessage(
|
||||
channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
content=prompt_text,
|
||||
metadata=msg.metadata,
|
||||
)
|
||||
)
|
||||
|
||||
# Wait for user reply
|
||||
pending = _PendingInterrupt(
|
||||
thread_id=thread_id,
|
||||
action_requests=action_reqs,
|
||||
event=asyncio.Event(),
|
||||
)
|
||||
self._pending_interrupts[session_key] = pending
|
||||
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
pending.event.wait(),
|
||||
timeout=_HITL_APPROVAL_TIMEOUT,
|
||||
)
|
||||
except TimeoutError:
|
||||
# Auto-approve on timeout
|
||||
pending.decision = "approve"
|
||||
finally:
|
||||
self._pending_interrupts.pop(session_key, None)
|
||||
|
||||
decision = pending.decision or "approve"
|
||||
|
||||
if decision == "reject":
|
||||
await self.bus.publish_outbound(
|
||||
OutboundMessage(
|
||||
channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
content="Tool execution rejected.",
|
||||
metadata=msg.metadata,
|
||||
)
|
||||
)
|
||||
return
|
||||
|
||||
if decision == "auto":
|
||||
self._auto_approve_sessions.add(session_key)
|
||||
|
||||
from langgraph.types import Command # type: ignore[import-untyped]
|
||||
|
||||
stream_input = Command(
|
||||
resume={"decisions": [{"type": "approve"} for _ in range(n)]}
|
||||
)
|
||||
# continue to next HITL round
|
||||
|
||||
except TimeoutError:
|
||||
self._metrics.total_timeouts += 1
|
||||
elapsed = time.monotonic() - _t0
|
||||
logger.error(
|
||||
"Inference timeout (%ds idle) for %s in %s, elapsed=%.2fs, response=%d chars",
|
||||
self._inference_timeout,
|
||||
msg.sender_id,
|
||||
session_key,
|
||||
elapsed,
|
||||
len(final_content),
|
||||
)
|
||||
await self.bus.publish_outbound(
|
||||
OutboundMessage(
|
||||
channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
content="Sorry, the response timed out. Please try again.",
|
||||
metadata=msg.metadata,
|
||||
)
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
self._metrics.total_failures += 1
|
||||
elapsed = time.monotonic() - _t0
|
||||
logger.error(
|
||||
"Agent error: %s | elapsed=%.2fs, response=%d chars, session=%s",
|
||||
e,
|
||||
elapsed,
|
||||
len(final_content),
|
||||
session_key,
|
||||
)
|
||||
await self.bus.publish_outbound(
|
||||
OutboundMessage(
|
||||
channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
content="Sorry, something went wrong. Please try again later.",
|
||||
metadata=msg.metadata,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
if channel:
|
||||
await channel.stop_typing(msg.chat_id)
|
||||
|
||||
# ── observability ──
|
||||
|
||||
@property
|
||||
def pending_count(self) -> int:
|
||||
"""Number of messages waiting in the work queue."""
|
||||
return self._work_queue.qsize()
|
||||
|
||||
@property
|
||||
def active_workers(self) -> int:
|
||||
"""Number of worker tasks that are still alive."""
|
||||
return sum(1 for w in self._workers if not w.done())
|
||||
|
||||
@property
|
||||
def metrics(self) -> dict[str, int]:
|
||||
"""Cumulative processing counters."""
|
||||
m = self._metrics
|
||||
return {
|
||||
"total_processed": m.total_processed,
|
||||
"total_successes": m.total_successes,
|
||||
"total_failures": m.total_failures,
|
||||
"total_timeouts": m.total_timeouts,
|
||||
"pending": self.pending_count,
|
||||
"active_workers": self.active_workers,
|
||||
"chat_locks": len(self._chat_locks),
|
||||
"sessions": len(self._sessions),
|
||||
}
|
||||
|
||||
# ── ask_user helpers ──
|
||||
|
||||
async def _wait_for_ask_user_reply(
|
||||
self,
|
||||
session_key: str,
|
||||
timeout: float,
|
||||
) -> str | None:
|
||||
"""Register a pending ask_user slot and wait for the user to reply.
|
||||
|
||||
Returns the raw reply text, or ``None`` on timeout.
|
||||
"""
|
||||
pending = _PendingAskUserReply(event=asyncio.Event())
|
||||
self._pending_ask_user_replies[session_key] = pending
|
||||
try:
|
||||
await asyncio.wait_for(pending.event.wait(), timeout=timeout)
|
||||
except TimeoutError:
|
||||
pass
|
||||
finally:
|
||||
self._pending_ask_user_replies.pop(session_key, None)
|
||||
return pending.reply
|
||||
|
||||
async def _resolve_ask_user(
|
||||
self,
|
||||
msg: InboundMessage,
|
||||
event_data: dict,
|
||||
session_key: str,
|
||||
) -> dict:
|
||||
"""Handle an ask_user interrupt: send questions to channel, collect answers.
|
||||
|
||||
Mirrors the logic of ``cli.channel.channel_ask_user_prompt`` but runs
|
||||
fully async inside the consumer event loop.
|
||||
|
||||
Returns a dict suitable for ``Command(resume=...)``:
|
||||
``{"answers": [...], "status": "answered"}`` or
|
||||
``{"status": "cancelled"}``.
|
||||
"""
|
||||
questions = event_data.get("questions", [])
|
||||
if not questions:
|
||||
return {"answers": [], "status": "answered"}
|
||||
|
||||
total = len(questions)
|
||||
answers: list[str] = []
|
||||
|
||||
for i, q in enumerate(questions):
|
||||
q_text = q.get("question", "")
|
||||
q_type = q.get("type", "text")
|
||||
required = q.get("required", True)
|
||||
|
||||
# -- Format question header --
|
||||
if total == 1:
|
||||
header = "\u2753 Quick check-in from EvoScientist\n"
|
||||
else:
|
||||
header = f"\u2753 Question {i + 1}/{total}\n"
|
||||
|
||||
lines: list[str] = [header, f"{i + 1}. {q_text}"]
|
||||
if not required:
|
||||
lines[-1] += " (optional)"
|
||||
|
||||
if q_type == "multiple_choice":
|
||||
choices = q.get("choices", [])
|
||||
for j, choice in enumerate(choices):
|
||||
label = choice.get("value", str(choice))
|
||||
letter = chr(ord("A") + j)
|
||||
lines.append(f" {letter}. {label}")
|
||||
other_letter = chr(ord("A") + len(choices))
|
||||
lines.append(f" {other_letter}. Other")
|
||||
letters = "/".join(chr(ord("A") + k) for k in range(len(choices) + 1))
|
||||
lines.append(f"\nReply with a letter ({letters}), or 'cancel'.")
|
||||
else:
|
||||
skip_hint = " Leave empty to skip." if not required else ""
|
||||
lines.append(f"\nReply with your answer, or 'cancel'.{skip_hint}")
|
||||
|
||||
# -- Send question --
|
||||
await self.bus.publish_outbound(
|
||||
OutboundMessage(
|
||||
channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
content="\n".join(lines),
|
||||
metadata=msg.metadata,
|
||||
)
|
||||
)
|
||||
|
||||
# -- Wait for user reply --
|
||||
reply = await self._wait_for_ask_user_reply(
|
||||
session_key,
|
||||
_ASK_USER_TIMEOUT,
|
||||
)
|
||||
|
||||
if not reply:
|
||||
await self.bus.publish_outbound(
|
||||
OutboundMessage(
|
||||
channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
content="\u23f0 Response timed out.",
|
||||
metadata=msg.metadata,
|
||||
)
|
||||
)
|
||||
return {"status": "cancelled"}
|
||||
|
||||
raw = reply.strip()
|
||||
if raw.lower() == "cancel":
|
||||
return {"status": "cancelled"}
|
||||
|
||||
# -- Parse answer --
|
||||
if q_type == "multiple_choice":
|
||||
choices = q.get("choices", [])
|
||||
other_letter = chr(ord("A") + len(choices))
|
||||
if len(raw) == 1 and raw.upper() == other_letter:
|
||||
# "Other" selected — ask for free-form input
|
||||
await self.bus.publish_outbound(
|
||||
OutboundMessage(
|
||||
channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
content="Please type your answer:",
|
||||
metadata=msg.metadata,
|
||||
)
|
||||
)
|
||||
other_reply = await self._wait_for_ask_user_reply(
|
||||
session_key,
|
||||
_ASK_USER_TIMEOUT,
|
||||
)
|
||||
if not other_reply:
|
||||
await self.bus.publish_outbound(
|
||||
OutboundMessage(
|
||||
channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
content="\u23f0 Response timed out.",
|
||||
metadata=msg.metadata,
|
||||
)
|
||||
)
|
||||
return {"status": "cancelled"}
|
||||
if other_reply.strip().lower() == "cancel":
|
||||
return {"status": "cancelled"}
|
||||
answers.append(other_reply.strip())
|
||||
elif len(raw) == 1 and raw.upper().isalpha():
|
||||
idx = ord(raw.upper()) - ord("A")
|
||||
if 0 <= idx < len(choices):
|
||||
answers.append(choices[idx].get("value", raw))
|
||||
else:
|
||||
answers.append(raw)
|
||||
else:
|
||||
answers.append(raw)
|
||||
else:
|
||||
answers.append(raw)
|
||||
|
||||
return {"answers": answers, "status": "answered"}
|
||||
|
||||
# ── internal ──
|
||||
|
||||
def _evict_chat_locks(self) -> None:
|
||||
"""Remove chat locks that are not currently held."""
|
||||
stale = [k for k, lock in self._chat_locks.items() if not lock.locked()]
|
||||
for k in stale[: max(1, len(stale) // 2)]:
|
||||
del self._chat_locks[k]
|
||||
@@ -0,0 +1,174 @@
|
||||
"""Shared debug logging helpers for channel integrations.
|
||||
|
||||
This module is intentionally channel-agnostic. Future per-channel PRs should
|
||||
reuse these helpers instead of reintroducing ad-hoc logging formats or
|
||||
standalone ``basicConfig`` calls.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
_REDACTED = "***"
|
||||
_SECRET_TOKENS = (
|
||||
"token",
|
||||
"secret",
|
||||
"password",
|
||||
"authorization",
|
||||
"cookie",
|
||||
"api_key",
|
||||
"apikey",
|
||||
"access_key",
|
||||
"private_key",
|
||||
"signature",
|
||||
)
|
||||
|
||||
|
||||
def _load_debug_trace_flag() -> bool:
|
||||
"""Load the trace feature switch from config as a fallback to env vars."""
|
||||
try:
|
||||
from ..config.settings import load_config
|
||||
|
||||
return bool(getattr(load_config(), "channel_debug_tracing", False))
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def debug_trace_enabled(enabled: bool | None = None) -> bool:
|
||||
"""Resolve the channel debug tracing switch.
|
||||
|
||||
Explicit ``enabled`` takes precedence; otherwise the helper falls back to
|
||||
``EVOSCIENTIST_CHANNEL_DEBUG_TRACING``.
|
||||
"""
|
||||
|
||||
if enabled is not None:
|
||||
return bool(enabled)
|
||||
raw = os.environ.get("EVOSCIENTIST_CHANNEL_DEBUG_TRACING", "")
|
||||
if raw.strip():
|
||||
return raw.strip().lower() in {"1", "true", "yes", "on"}
|
||||
return _load_debug_trace_flag()
|
||||
|
||||
|
||||
def _should_redact_key(key: str) -> bool:
|
||||
lowered = key.lower()
|
||||
return any(token in lowered for token in _SECRET_TOKENS)
|
||||
|
||||
|
||||
def _stringify(value: Any) -> str:
|
||||
if value is None:
|
||||
return "-"
|
||||
if isinstance(value, bool):
|
||||
return "true" if value else "false"
|
||||
if isinstance(value, bytes):
|
||||
return f"<bytes:{len(value)}>"
|
||||
if isinstance(value, str):
|
||||
return value.replace("\n", "\\n")
|
||||
if isinstance(value, Mapping):
|
||||
return f"<map:{len(value)}>"
|
||||
if isinstance(value, Sequence) and not isinstance(value, str | bytes | bytearray):
|
||||
return f"<seq:{len(value)}>"
|
||||
return str(value).replace("\n", "\\n")
|
||||
|
||||
|
||||
def _format_fields(fields: Mapping[str, Any]) -> str:
|
||||
parts: list[str] = []
|
||||
for key, value in fields.items():
|
||||
if value is None:
|
||||
continue
|
||||
safe_value = _REDACTED if _should_redact_key(key) else _stringify(value)
|
||||
parts.append(f"{key}={safe_value}")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
_warned_debug_level_mismatch = False
|
||||
|
||||
|
||||
def _warn_debug_level_mismatch(logger: logging.Logger) -> None:
|
||||
"""Emit a one-time warning when tracing is enabled but DEBUG logs are hidden."""
|
||||
global _warned_debug_level_mismatch
|
||||
if _warned_debug_level_mismatch:
|
||||
return
|
||||
_warned_debug_level_mismatch = True
|
||||
logger.warning(
|
||||
"channel debug tracing is enabled but logger level is above DEBUG; "
|
||||
"set EVOSCIENTIST_LOG_LEVEL=DEBUG, configure log_level=debug, "
|
||||
"or use 'serve --debug' to see trace events"
|
||||
)
|
||||
|
||||
|
||||
def emit_debug_event(
|
||||
logger: logging.Logger,
|
||||
event: str,
|
||||
*,
|
||||
channel: str,
|
||||
enabled: bool,
|
||||
**fields: Any,
|
||||
) -> None:
|
||||
"""Emit a structured channel debug event.
|
||||
|
||||
Example output:
|
||||
``event=inbound_raw channel=telegram message_id=123 chat_id=-1001``
|
||||
"""
|
||||
|
||||
if not enabled:
|
||||
return
|
||||
if not logger.isEnabledFor(logging.DEBUG):
|
||||
_warn_debug_level_mismatch(logger)
|
||||
return
|
||||
base_fields = {"event": event, "channel": channel}
|
||||
base_fields.update(fields)
|
||||
logger.debug(_format_fields(base_fields))
|
||||
|
||||
|
||||
def emit_debug_event_if(
|
||||
logger: logging.Logger,
|
||||
event: str,
|
||||
enabled: bool,
|
||||
**fields: Any,
|
||||
) -> None:
|
||||
"""Convenience wrapper for code that lacks a :class:`Channel` instance.
|
||||
|
||||
Unlike :func:`emit_debug_event`, the ``channel`` field is not required —
|
||||
pass it via *fields* when available. This is intended for middleware
|
||||
classes, managers, and standalone helpers.
|
||||
"""
|
||||
|
||||
if not enabled:
|
||||
return
|
||||
if not logger.isEnabledFor(logging.DEBUG):
|
||||
_warn_debug_level_mismatch(logger)
|
||||
return
|
||||
base_fields: dict[str, Any] = {"event": event}
|
||||
base_fields.update(fields)
|
||||
logger.debug(_format_fields(base_fields))
|
||||
|
||||
|
||||
class TraceMixin:
|
||||
"""Mixin providing unified structured trace helpers.
|
||||
|
||||
Classes using this mixin must set ``_debug_trace`` (bool) and
|
||||
``_trace_logger`` attribute returning a :class:`logging.Logger`.
|
||||
|
||||
The trace name defaults to ``self.name`` if present, otherwise
|
||||
``"unknown"``. Override ``_trace_name`` to customise.
|
||||
"""
|
||||
|
||||
_debug_trace: bool
|
||||
_trace_logger: logging.Logger
|
||||
|
||||
@property
|
||||
def _trace_name(self) -> str:
|
||||
return getattr(self, "name", "unknown")
|
||||
|
||||
def _trace_event(self, event: str, **fields: Any) -> None:
|
||||
"""Emit a structured debug event when tracing is enabled."""
|
||||
emit_debug_event(
|
||||
self._trace_logger,
|
||||
event,
|
||||
channel=self._trace_name,
|
||||
enabled=self._debug_trace,
|
||||
**fields,
|
||||
)
|
||||
@@ -0,0 +1,31 @@
|
||||
"""DingTalk (钉钉) channel for EvoScientist.
|
||||
|
||||
Uses Stream Mode (WebSocket) for receiving messages — no public IP needed.
|
||||
Sends replies via HTTP API.
|
||||
|
||||
Usage in config:
|
||||
channel_enabled = "dingtalk"
|
||||
dingtalk_client_id = "your_app_key"
|
||||
dingtalk_client_secret = "your_app_secret"
|
||||
"""
|
||||
|
||||
from ..channel_manager import _parse_csv, register_channel
|
||||
from .channel import DingTalkChannel, DingTalkConfig
|
||||
|
||||
__all__ = ["DingTalkChannel", "DingTalkConfig"]
|
||||
|
||||
|
||||
def create_from_config(config) -> DingTalkChannel:
|
||||
allowed = _parse_csv(config.dingtalk_allowed_senders)
|
||||
proxy = config.dingtalk_proxy or None
|
||||
return DingTalkChannel(
|
||||
DingTalkConfig(
|
||||
client_id=config.dingtalk_client_id,
|
||||
client_secret=config.dingtalk_client_secret,
|
||||
allowed_senders=allowed,
|
||||
proxy=proxy,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
register_channel("dingtalk", create_from_config)
|
||||
@@ -0,0 +1,437 @@
|
||||
"""DingTalk channel — refactored with WebSocketMixin + TokenMixin."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import ClassVar
|
||||
from urllib.parse import quote_plus
|
||||
|
||||
from ..base import Channel, ChannelError, RawIncoming
|
||||
from ..capabilities import DINGTALK as DINGTALK_CAPS
|
||||
from ..config import BaseChannelConfig
|
||||
from ..mixins import TokenMixin, WebSocketMixin
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
GATEWAY_URL = "https://api.dingtalk.com/v1.0/gateway/connections/open"
|
||||
TOKEN_URL = "https://api.dingtalk.com/v1.0/oauth2/accessToken"
|
||||
SEND_URL = "https://api.dingtalk.com/v1.0/robot/oToMessages/batchSend"
|
||||
MEDIA_SEND_URL = "https://api.dingtalk.com/v1.0/robot/oToMessages/batchSend"
|
||||
MEDIA_UPLOAD_URL = "https://oapi.dingtalk.com/media/upload"
|
||||
FILE_DOWNLOAD_URL = "https://api.dingtalk.com/v1.0/robot/messageFiles/download"
|
||||
|
||||
|
||||
@dataclass
|
||||
class DingTalkConfig(BaseChannelConfig):
|
||||
client_id: str = ""
|
||||
client_secret: str = ""
|
||||
text_chunk_limit: int = 4096
|
||||
|
||||
|
||||
class DingTalkChannel(Channel, WebSocketMixin, TokenMixin):
|
||||
capabilities = DINGTALK_CAPS
|
||||
name = "dingtalk"
|
||||
_ready_attrs = ("_http_client", "_access_token")
|
||||
_non_retryable_patterns = ("invalidauthentication", "forbidden", "40014")
|
||||
_mention_pattern = r"@\S+\s*"
|
||||
_mention_strip_count = 1
|
||||
|
||||
def __init__(self, config: DingTalkConfig):
|
||||
super().__init__(config)
|
||||
|
||||
async def start(self) -> None:
|
||||
import httpx
|
||||
|
||||
if not self.config.client_id or not self.config.client_secret:
|
||||
raise ChannelError("DingTalk client_id and client_secret are required")
|
||||
self._http_client = httpx.AsyncClient(timeout=15, proxy=self.config.proxy)
|
||||
await self._refresh_token()
|
||||
self._running = True
|
||||
logger.info("DingTalk channel starting (Stream Mode)...")
|
||||
self._ws_task = asyncio.create_task(self._ws_loop())
|
||||
|
||||
# ── TokenMixin ────────────────────────────────────────────────
|
||||
|
||||
async def _fetch_token(self) -> tuple[str, int]:
|
||||
data = await self._api_post(
|
||||
TOKEN_URL,
|
||||
{
|
||||
"appKey": self.config.client_id,
|
||||
"appSecret": self.config.client_secret,
|
||||
},
|
||||
)
|
||||
token = data.get("accessToken")
|
||||
if not token:
|
||||
raise ChannelError(f"DingTalk auth error: {data}")
|
||||
return token, int(data.get("expireIn", 7200))
|
||||
|
||||
async def _api_post(self, url, body, headers=None):
|
||||
resp = await self._http_client.post(url, json=body, headers=headers)
|
||||
return resp.json()
|
||||
|
||||
async def _resolve_download_code(self, download_code: str) -> str | None:
|
||||
"""Exchange a DingTalk downloadCode for a real download URL."""
|
||||
try:
|
||||
token = await self._ensure_token()
|
||||
data = await self._api_post(
|
||||
FILE_DOWNLOAD_URL,
|
||||
{"downloadCode": download_code, "robotCode": self.config.client_id},
|
||||
headers={"x-acs-dingtalk-access-token": token},
|
||||
)
|
||||
url = data.get("downloadUrl") or ""
|
||||
if url:
|
||||
return url
|
||||
logger.warning(f"DingTalk downloadCode resolve failed: {data}")
|
||||
except Exception as e:
|
||||
logger.warning(f"DingTalk downloadCode resolve error: {e}")
|
||||
return None
|
||||
|
||||
# ── WebSocketMixin ────────────────────────────────────────────
|
||||
|
||||
async def _get_ws_url(self) -> str:
|
||||
resp = await self._http_client.post(
|
||||
GATEWAY_URL,
|
||||
json={
|
||||
"clientId": self.config.client_id,
|
||||
"clientSecret": self.config.client_secret,
|
||||
"subscriptions": [
|
||||
{"type": "CALLBACK", "topic": "/v1.0/im/bot/messages/get"}
|
||||
],
|
||||
"ua": "dingtalk-sdk-python/v0.24.3-union",
|
||||
},
|
||||
)
|
||||
data = resp.json()
|
||||
endpoint, ticket = data.get("endpoint"), data.get("ticket")
|
||||
if not endpoint or not ticket:
|
||||
raise ChannelError(f"DingTalk gateway failed: {data}")
|
||||
return f"{endpoint}?ticket={quote_plus(ticket)}"
|
||||
|
||||
async def _on_ws_message(self, data) -> None:
|
||||
if not isinstance(data, dict):
|
||||
return
|
||||
headers = data.get("headers", {})
|
||||
msg_id = headers.get("messageId", "")
|
||||
|
||||
# System ping
|
||||
if data.get("type") == "SYSTEM" and headers.get("topic") == "ping":
|
||||
await self._ws_send_json(
|
||||
{
|
||||
"code": 200,
|
||||
"headers": headers,
|
||||
"message": "OK",
|
||||
"data": data.get("data", ""),
|
||||
}
|
||||
)
|
||||
return
|
||||
|
||||
# ACK
|
||||
await self._ws_send_json(
|
||||
{
|
||||
"code": 200,
|
||||
"headers": {"contentType": "application/json", "messageId": msg_id},
|
||||
"message": "OK",
|
||||
"data": "{}",
|
||||
}
|
||||
)
|
||||
|
||||
if data.get("type") != "CALLBACK":
|
||||
return
|
||||
|
||||
payload = data.get("data", "{}")
|
||||
payload = json.loads(payload) if isinstance(payload, str) else payload
|
||||
text_obj = payload.get("text", {})
|
||||
content = (
|
||||
text_obj.get("content", "") if isinstance(text_obj, dict) else str(text_obj)
|
||||
).strip()
|
||||
if not content:
|
||||
raw_content = payload.get("content", "")
|
||||
content = raw_content.strip() if isinstance(raw_content, str) else ""
|
||||
|
||||
# Download attachments if present
|
||||
annotations: list[str] = []
|
||||
media_paths: list[str] = []
|
||||
|
||||
# DingTalk file/image messages may put download info in
|
||||
# payload["content"] (as a dict) instead of in a dedicated
|
||||
# "fileContent"/"imageContent" key.
|
||||
raw_content_obj = payload.get("content")
|
||||
if isinstance(raw_content_obj, dict) and raw_content_obj not in [
|
||||
payload.get(k)
|
||||
for k in ("imageContent", "fileContent", "videoContent", "audioContent")
|
||||
]:
|
||||
msg_type = payload.get("msgtype") or payload.get("msgType") or ""
|
||||
media_label = msg_type or "file"
|
||||
file_size = (
|
||||
raw_content_obj.get("fileSize")
|
||||
or raw_content_obj.get("downloadSize")
|
||||
or 0
|
||||
)
|
||||
file_name = (
|
||||
raw_content_obj.get("fileName")
|
||||
or raw_content_obj.get("name")
|
||||
or f"dingtalk_{msg_type}"
|
||||
)
|
||||
download_code = raw_content_obj.get("downloadCode") or ""
|
||||
download_url = raw_content_obj.get("downloadUrl") or ""
|
||||
# downloadCode is NOT a URL — resolve it via DingTalk API first
|
||||
if download_code and not download_code.startswith("http"):
|
||||
resolved = await self._resolve_download_code(download_code)
|
||||
if resolved:
|
||||
download_url = resolved
|
||||
elif download_code:
|
||||
download_url = download_code
|
||||
if download_url:
|
||||
try:
|
||||
dl_token = await self._ensure_token()
|
||||
dl_headers = {"x-acs-dingtalk-access-token": dl_token}
|
||||
except Exception:
|
||||
dl_headers = None
|
||||
local, ann = await self._download_attachment(
|
||||
download_url,
|
||||
f"dingtalk_{file_name}",
|
||||
headers=dl_headers,
|
||||
file_size=int(file_size) if file_size else None,
|
||||
)
|
||||
if local:
|
||||
media_paths.append(local)
|
||||
if ann:
|
||||
ann = ann.replace("[attachment:", f"[{media_label}:")
|
||||
annotations.append(ann)
|
||||
elif file_name:
|
||||
annotations.append(f"[{media_label}: {file_name}]")
|
||||
|
||||
for att_key in ("imageContent", "fileContent", "videoContent", "audioContent"):
|
||||
att = payload.get(att_key)
|
||||
if att and isinstance(att, dict):
|
||||
file_size = att.get("fileSize") or att.get("downloadSize") or 0
|
||||
file_name = att.get("fileName", att_key)
|
||||
download_code = att.get("downloadCode") or ""
|
||||
download_url = att.get("downloadUrl") or ""
|
||||
# Resolve downloadCode via API if it's not a URL
|
||||
if download_code and not download_code.startswith("http"):
|
||||
resolved = await self._resolve_download_code(download_code)
|
||||
if resolved:
|
||||
download_url = resolved
|
||||
elif download_code:
|
||||
download_url = download_code
|
||||
# DingTalk audioContent is voice messages
|
||||
media_label = "voice" if att_key == "audioContent" else att_key
|
||||
if download_url and (
|
||||
self.config.include_attachments
|
||||
if hasattr(self.config, "include_attachments")
|
||||
else True
|
||||
):
|
||||
# DingTalk download URLs require access token
|
||||
try:
|
||||
dl_token = await self._ensure_token()
|
||||
dl_headers = {"x-acs-dingtalk-access-token": dl_token}
|
||||
except Exception:
|
||||
dl_headers = None
|
||||
local, ann = await self._download_attachment(
|
||||
download_url,
|
||||
f"dingtalk_{file_name}",
|
||||
headers=dl_headers,
|
||||
file_size=int(file_size) if file_size else None,
|
||||
)
|
||||
if local:
|
||||
media_paths.append(local)
|
||||
if ann:
|
||||
ann = ann.replace("[attachment:", f"[{media_label}:")
|
||||
annotations.append(ann)
|
||||
elif file_size:
|
||||
too_large = self._check_attachment_size(int(file_size), file_name)
|
||||
if too_large:
|
||||
annotations.append(too_large)
|
||||
else:
|
||||
annotations.append(f"[{media_label}: {file_name}]")
|
||||
|
||||
if not content and not media_paths and not annotations:
|
||||
return
|
||||
|
||||
sender_id = payload.get("senderStaffId") or payload.get("senderId", "")
|
||||
is_group = payload.get("conversationType") == "2"
|
||||
# For send API (oToMessages/batchSend), userIds needs staffId, not conversationId
|
||||
chat_id = sender_id
|
||||
create_time = payload.get("createAt") or payload.get("createTime", "")
|
||||
|
||||
# Mention gating: DMs always pass; groups require @bot
|
||||
was_mentioned = not is_group
|
||||
if is_group:
|
||||
# isInAtList is set by DingTalk when bot is @mentioned
|
||||
if payload.get("isInAtList"):
|
||||
was_mentioned = True
|
||||
else:
|
||||
# Fallback: check atUsers array
|
||||
at_users = payload.get("atUsers") or []
|
||||
for u in at_users:
|
||||
if u.get("dingtalkId") == self.config.client_id:
|
||||
was_mentioned = True
|
||||
break
|
||||
|
||||
try:
|
||||
ts = (
|
||||
datetime.fromtimestamp(int(create_time) / 1000)
|
||||
if create_time
|
||||
else datetime.now()
|
||||
)
|
||||
except (ValueError, TypeError, OSError):
|
||||
ts = datetime.now()
|
||||
|
||||
await self._enqueue_raw(
|
||||
RawIncoming(
|
||||
sender_id=sender_id,
|
||||
chat_id=chat_id,
|
||||
text=content,
|
||||
timestamp=ts,
|
||||
message_id=msg_id,
|
||||
is_group=is_group,
|
||||
was_mentioned=was_mentioned,
|
||||
media_files=media_paths,
|
||||
content_annotations=annotations,
|
||||
metadata={
|
||||
"chat_id": chat_id,
|
||||
"sender_nick": payload.get("senderNick", ""),
|
||||
"backend": "dingtalk",
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
# _send_typing_action: inherited no-op (DingTalk has no typing API)
|
||||
# _format_chunk: inherited from base (UnifiedFormatter)
|
||||
|
||||
# ── Send ──────────────────────────────────────────────────────
|
||||
|
||||
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
|
||||
token = await self._ensure_token()
|
||||
data = await self._api_post(
|
||||
SEND_URL,
|
||||
{
|
||||
"robotCode": self.config.client_id,
|
||||
"userIds": [chat_id],
|
||||
"msgKey": "sampleMarkdown",
|
||||
"msgParam": json.dumps({"text": raw_text, "title": "EvoScientist"}),
|
||||
},
|
||||
headers={"x-acs-dingtalk-access-token": token},
|
||||
)
|
||||
return data
|
||||
|
||||
# ── Media send ────────────────────────────────────────────────
|
||||
|
||||
_IMAGE_EXTS: ClassVar[set[str]] = {".jpg", ".jpeg", ".png", ".gif", ".bmp", ".webp"}
|
||||
|
||||
async def _send_media_impl(
|
||||
self,
|
||||
recipient: str,
|
||||
file_path: str,
|
||||
caption: str = "",
|
||||
metadata: dict | None = None,
|
||||
) -> bool:
|
||||
"""Send a media file through DingTalk.
|
||||
|
||||
For images: uploads via /media/upload to get media_id, then sends
|
||||
as sampleImageMsg. Non-image files are sent as markdown links
|
||||
(DingTalk robot API does not support arbitrary file uploads).
|
||||
"""
|
||||
token = await self._ensure_token()
|
||||
chat_id = self._resolve_media_chat_id(recipient, metadata)
|
||||
headers = {"x-acs-dingtalk-access-token": token}
|
||||
ext = Path(file_path).suffix.lower()
|
||||
|
||||
if ext in self._IMAGE_EXTS:
|
||||
# Try uploading image to get media_id for native image message
|
||||
media_id = await self._upload_dingtalk_media(token, file_path, "image")
|
||||
if media_id:
|
||||
await self._api_post(
|
||||
MEDIA_SEND_URL,
|
||||
{
|
||||
"robotCode": self.config.client_id,
|
||||
"userIds": [chat_id],
|
||||
"msgKey": "sampleImageMsg",
|
||||
"msgParam": json.dumps({"photoURL": media_id}),
|
||||
},
|
||||
headers=headers,
|
||||
)
|
||||
else:
|
||||
# Fallback to markdown with file path
|
||||
await self._api_post(
|
||||
MEDIA_SEND_URL,
|
||||
{
|
||||
"robotCode": self.config.client_id,
|
||||
"userIds": [chat_id],
|
||||
"msgKey": "sampleMarkdown",
|
||||
"msgParam": json.dumps(
|
||||
{
|
||||
"text": f""
|
||||
+ (f"\n{caption}" if caption else ""),
|
||||
"title": caption or "Image",
|
||||
}
|
||||
),
|
||||
},
|
||||
headers=headers,
|
||||
)
|
||||
else:
|
||||
# Non-image: send as markdown with filename
|
||||
name = Path(file_path).name
|
||||
text = f"[文件] {name}" + (f"\n{caption}" if caption else "")
|
||||
await self._api_post(
|
||||
MEDIA_SEND_URL,
|
||||
{
|
||||
"robotCode": self.config.client_id,
|
||||
"userIds": [chat_id],
|
||||
"msgKey": "sampleMarkdown",
|
||||
"msgParam": json.dumps({"text": text, "title": name}),
|
||||
},
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
if caption and ext in self._IMAGE_EXTS:
|
||||
# Send caption separately for image messages
|
||||
await self._api_post(
|
||||
MEDIA_SEND_URL,
|
||||
{
|
||||
"robotCode": self.config.client_id,
|
||||
"userIds": [chat_id],
|
||||
"msgKey": "sampleMarkdown",
|
||||
"msgParam": json.dumps({"text": caption, "title": "Caption"}),
|
||||
},
|
||||
headers=headers,
|
||||
)
|
||||
return True
|
||||
|
||||
async def _upload_dingtalk_media(
|
||||
self,
|
||||
token: str,
|
||||
file_path: str,
|
||||
media_type: str = "image",
|
||||
) -> str | None:
|
||||
"""Upload a file to DingTalk media API and return the media_id."""
|
||||
try:
|
||||
url = f"{MEDIA_UPLOAD_URL}?access_token={token}&type={media_type}"
|
||||
with open(file_path, "rb") as f:
|
||||
resp = await self._http_client.post(
|
||||
url,
|
||||
files={"media": (Path(file_path).name, f)},
|
||||
)
|
||||
data = resp.json()
|
||||
return data.get("media_id")
|
||||
except Exception as e:
|
||||
logger.warning(f"DingTalk media upload failed: {e}")
|
||||
return None
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
if hasattr(self, "_ws_task") and self._ws_task:
|
||||
self._ws_task.cancel()
|
||||
try:
|
||||
await self._ws_task
|
||||
except (asyncio.CancelledError, Exception):
|
||||
pass
|
||||
self._ws_task = None
|
||||
await self._stop_ws()
|
||||
if hasattr(self, "_http_client") and self._http_client:
|
||||
await self._http_client.aclose()
|
||||
self._http_client = None
|
||||
self._access_token = None
|
||||
logger.info("DingTalk channel stopped")
|
||||
@@ -0,0 +1,33 @@
|
||||
"""DingTalk credential validation."""
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def validate_dingtalk(
|
||||
client_id: str,
|
||||
client_secret: str,
|
||||
proxy: str | None = None,
|
||||
) -> tuple[bool, str]:
|
||||
"""Validate DingTalk credentials by fetching an access token."""
|
||||
if not client_id or not client_secret:
|
||||
return False, "client_id and client_secret are required"
|
||||
|
||||
try:
|
||||
import httpx
|
||||
except ImportError:
|
||||
return False, "httpx not installed"
|
||||
|
||||
url = "https://api.dingtalk.com/v1.0/oauth2/accessToken"
|
||||
body = {"appKey": client_id, "appSecret": client_secret}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(proxy=proxy) as client:
|
||||
resp = await client.post(url, json=body, timeout=10)
|
||||
data = resp.json()
|
||||
if data.get("accessToken"):
|
||||
return True, "DingTalk credentials valid"
|
||||
return False, f"Error: {data.get('message', data)}"
|
||||
except Exception as e:
|
||||
return False, f"Error: {e}"
|
||||
@@ -0,0 +1,89 @@
|
||||
"""DingTalk channel server.
|
||||
|
||||
Standalone script to run the DingTalk channel with CLI options.
|
||||
|
||||
Usage:
|
||||
python -m EvoScientist.channels.dingtalk.serve --client-id ID --client-secret SECRET [OPTIONS]
|
||||
|
||||
Examples:
|
||||
# Basic usage
|
||||
python -m EvoScientist.channels.dingtalk.serve --client-id ID --client-secret SECRET
|
||||
|
||||
# With proxy and allowed senders
|
||||
python -m EvoScientist.channels.dingtalk.serve --client-id ID --client-secret SECRET --proxy http://proxy:8080 --allow user123
|
||||
|
||||
# With agent and thinking
|
||||
python -m EvoScientist.channels.dingtalk.serve --client-id ID --client-secret SECRET --agent --thinking
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
from ...logging_config import configure_logging_from_settings
|
||||
from ..bus import MessageBus
|
||||
from ..standalone import run_standalone
|
||||
from .channel import DingTalkChannel, DingTalkConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="DingTalk channel server",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--client-id",
|
||||
required=True,
|
||||
help="DingTalk app client ID",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--client-secret",
|
||||
required=True,
|
||||
help="DingTalk app client secret",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow",
|
||||
action="append",
|
||||
dest="allowed_senders",
|
||||
help="Allowed sender (DingTalk user ID). Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--proxy",
|
||||
help="HTTP proxy URL",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--agent",
|
||||
action="store_true",
|
||||
help="Use EvoScientist agent as handler (default: echo)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--thinking",
|
||||
action="store_true",
|
||||
help="Send thinking content as intermediate messages (requires --agent)",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
"""Entry point."""
|
||||
configure_logging_from_settings(default_level=logging.INFO)
|
||||
args = parse_args()
|
||||
|
||||
config = DingTalkConfig(
|
||||
client_id=args.client_id,
|
||||
client_secret=args.client_secret,
|
||||
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
|
||||
proxy=args.proxy,
|
||||
)
|
||||
|
||||
send_thinking = args.thinking and args.agent
|
||||
bus = MessageBus()
|
||||
channel = DingTalkChannel(config)
|
||||
|
||||
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,21 @@
|
||||
from ..channel_manager import _parse_csv, register_channel
|
||||
from .channel import DiscordChannel, DiscordConfig
|
||||
|
||||
__all__ = ["DiscordChannel", "DiscordConfig"]
|
||||
|
||||
|
||||
def create_from_config(config) -> DiscordChannel:
|
||||
allowed = _parse_csv(config.discord_allowed_senders)
|
||||
channels = _parse_csv(config.discord_allowed_channels)
|
||||
proxy = config.discord_proxy or None
|
||||
return DiscordChannel(
|
||||
DiscordConfig(
|
||||
bot_token=config.discord_bot_token,
|
||||
allowed_senders=allowed,
|
||||
allowed_channels=channels,
|
||||
proxy=proxy,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
register_channel("discord", create_from_config)
|
||||
@@ -0,0 +1,274 @@
|
||||
"""Discord channel implementation using discord.py."""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
|
||||
from ..base import Channel, ChannelError, RawIncoming
|
||||
from ..capabilities import DISCORD as DISCORD_CAPS
|
||||
from ..config import BaseChannelConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DiscordConfig(BaseChannelConfig):
|
||||
bot_token: str = ""
|
||||
text_chunk_limit: int = 2000
|
||||
|
||||
|
||||
class DiscordChannel(Channel):
|
||||
"""Discord channel using discord.py."""
|
||||
|
||||
name = "discord"
|
||||
|
||||
capabilities = DISCORD_CAPS
|
||||
_typing_interval: float = 8.0
|
||||
_ready_attrs = ("_client",)
|
||||
_mention_pattern = r"<@!?{bot_id}>\s*"
|
||||
|
||||
def __init__(self, config: DiscordConfig):
|
||||
super().__init__(config)
|
||||
self._client = None
|
||||
self._ready = asyncio.Event()
|
||||
# Cache message objects for ACK reactions
|
||||
self._message_cache: dict[str, object] = {}
|
||||
self._MESSAGE_CACHE_MAX = 200
|
||||
self._background_tasks: set[asyncio.Task] = set()
|
||||
|
||||
async def start(self) -> None:
|
||||
try:
|
||||
import discord
|
||||
except ImportError:
|
||||
raise ChannelError(
|
||||
"discord.py not installed. "
|
||||
"Install with: pip install evoscientist[discord]"
|
||||
) from None
|
||||
|
||||
if not self.config.bot_token:
|
||||
raise ChannelError("Discord bot token is required")
|
||||
|
||||
proxy = (
|
||||
self.config.proxy
|
||||
or os.environ.get("https_proxy")
|
||||
or os.environ.get("HTTPS_PROXY")
|
||||
or os.environ.get("http_proxy")
|
||||
or os.environ.get("HTTP_PROXY")
|
||||
or None
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Discord connect: token=%s...%s proxy=%s",
|
||||
self.config.bot_token[:8],
|
||||
self.config.bot_token[-4:],
|
||||
proxy or "(none)",
|
||||
)
|
||||
|
||||
intents = discord.Intents.default()
|
||||
intents.message_content = True
|
||||
client_kwargs = {"intents": intents}
|
||||
if proxy:
|
||||
client_kwargs["proxy"] = proxy
|
||||
self._client = discord.Client(**client_kwargs)
|
||||
|
||||
self._start_task_error: BaseException | None = None
|
||||
|
||||
@self._client.event
|
||||
async def on_ready():
|
||||
logger.info(f"Discord bot ready: {self._client.user}")
|
||||
self._ready.set()
|
||||
|
||||
@self._client.event
|
||||
async def on_message(message):
|
||||
await self._on_message(message)
|
||||
|
||||
async def _guarded_start():
|
||||
try:
|
||||
logger.info("Discord gateway: starting client.start()...")
|
||||
await self._client.start(self.config.bot_token)
|
||||
except Exception as exc:
|
||||
logger.error("Discord gateway error: %s: %s", type(exc).__name__, exc)
|
||||
self._start_task_error = exc
|
||||
self._ready.set() # unblock the waiter so it doesn't hang
|
||||
|
||||
logger.info("Discord connect: launching gateway task")
|
||||
_task = asyncio.create_task(_guarded_start())
|
||||
self._background_tasks.add(_task)
|
||||
_task.add_done_callback(self._background_tasks.discard)
|
||||
|
||||
try:
|
||||
await asyncio.wait_for(self._ready.wait(), timeout=60)
|
||||
except TimeoutError:
|
||||
raise ChannelError(
|
||||
"Discord bot failed to connect within 60s. "
|
||||
"Check network/proxy connectivity to gateway.discord.gg"
|
||||
) from None
|
||||
|
||||
if self._start_task_error:
|
||||
raise ChannelError(
|
||||
f"Discord bot failed to connect: {self._start_task_error}"
|
||||
)
|
||||
|
||||
self._running = True
|
||||
logger.info("Discord channel started")
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
if self._client:
|
||||
await self._client.close()
|
||||
logger.info("Discord channel stopped")
|
||||
|
||||
# ── Typing indicator ────────────────────────────────────────────
|
||||
|
||||
async def _send_typing_action(self, chat_id: str) -> None:
|
||||
if not self._client:
|
||||
return
|
||||
ch = self._client.get_channel(int(chat_id))
|
||||
if ch:
|
||||
await ch.trigger_typing()
|
||||
|
||||
# ── ACK Reactions ───────────────────────────────────────────────
|
||||
|
||||
async def _send_ack_reaction(
|
||||
self, chat_id: str, message_id: str, emoji: str = "👀"
|
||||
) -> None:
|
||||
msg = self._message_cache.get(message_id)
|
||||
if msg:
|
||||
try:
|
||||
await msg.add_reaction(emoji)
|
||||
except Exception as e:
|
||||
logger.debug(f"Discord ACK reaction failed: {e}")
|
||||
|
||||
async def _remove_ack_reaction(
|
||||
self, chat_id: str, message_id: str, emoji: str = "👀"
|
||||
) -> None:
|
||||
msg = self._message_cache.get(message_id)
|
||||
if msg and self._client and self._client.user:
|
||||
try:
|
||||
await msg.remove_reaction(emoji, self._client.user)
|
||||
except Exception as e:
|
||||
logger.debug(f"Discord remove ACK reaction failed: {e}")
|
||||
|
||||
def _cache_message(self, message) -> None:
|
||||
"""Cache a discord message object for later reaction use."""
|
||||
mid = str(message.id)
|
||||
self._message_cache[mid] = message
|
||||
# Evict oldest entries if cache is too large
|
||||
if len(self._message_cache) > self._MESSAGE_CACHE_MAX:
|
||||
oldest = list(self._message_cache.keys())[: self._MESSAGE_CACHE_MAX // 2]
|
||||
for k in oldest:
|
||||
self._message_cache.pop(k, None)
|
||||
|
||||
# ── Send ────────────────────────────────────────────────────────
|
||||
|
||||
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
|
||||
import discord
|
||||
|
||||
thread_id = (metadata or {}).get("thread_id", "")
|
||||
target_id = int(thread_id) if thread_id else int(chat_id)
|
||||
ch = self._client.get_channel(target_id)
|
||||
if not ch:
|
||||
raise RuntimeError(f"Discord channel {target_id} not found")
|
||||
ref = None
|
||||
if reply_to:
|
||||
try:
|
||||
ref = discord.MessageReference(
|
||||
message_id=int(reply_to),
|
||||
channel_id=target_id,
|
||||
)
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
async def _send(text):
|
||||
await ch.send(text, reference=ref)
|
||||
|
||||
await self._send_with_format_fallback(_send, formatted_text, raw_text)
|
||||
|
||||
async def _send_media_impl(
|
||||
self,
|
||||
recipient: str,
|
||||
file_path: str,
|
||||
caption: str = "",
|
||||
metadata: dict | None = None,
|
||||
) -> bool:
|
||||
import discord
|
||||
|
||||
channel_id = self._resolve_media_chat_id(recipient, metadata)
|
||||
ch = self._client.get_channel(int(channel_id))
|
||||
if not ch:
|
||||
logger.error(f"Discord channel {channel_id} not found")
|
||||
return False
|
||||
file = discord.File(file_path)
|
||||
await ch.send(content=caption or None, file=file)
|
||||
return True
|
||||
|
||||
def _get_bot_identifier(self) -> str | None:
|
||||
if self._client and self._client.user:
|
||||
return str(self._client.user.id)
|
||||
return None
|
||||
|
||||
# ── Inbound ─────────────────────────────────────────────────────
|
||||
|
||||
async def _on_message(self, message) -> None:
|
||||
import discord
|
||||
|
||||
if message.author == self._client.user:
|
||||
return
|
||||
|
||||
# Cache for ACK reactions
|
||||
self._cache_message(message)
|
||||
|
||||
user_id = str(message.author.id)
|
||||
channel_id = str(message.channel.id)
|
||||
|
||||
is_dm = isinstance(message.channel, discord.DMChannel)
|
||||
was_mentioned = is_dm or (self._client.user in message.mentions)
|
||||
|
||||
text = message.content or ""
|
||||
annotations: list[str] = []
|
||||
media_paths: list[str] = []
|
||||
|
||||
if self.config.include_attachments and message.attachments:
|
||||
for attachment in message.attachments:
|
||||
too_large = self._check_attachment_size(
|
||||
attachment.size or 0,
|
||||
attachment.filename,
|
||||
)
|
||||
if too_large:
|
||||
annotations.append(too_large)
|
||||
continue
|
||||
try:
|
||||
safe_name = attachment.filename.replace("/", "_")
|
||||
file_path = self._media_path(f"{attachment.id}_{safe_name}")
|
||||
await attachment.save(file_path)
|
||||
media_paths.append(str(file_path))
|
||||
annotations.append(f"[attachment: {file_path}]")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to download Discord attachment: {e}")
|
||||
annotations.append(
|
||||
f"[attachment: {attachment.filename} - download failed]"
|
||||
)
|
||||
|
||||
# Detect thread context
|
||||
thread_id = ""
|
||||
parent_channel_id = channel_id
|
||||
if hasattr(message.channel, "parent") and message.channel.parent:
|
||||
# Message is inside a Thread — store thread info
|
||||
thread_id = channel_id # the thread IS the channel
|
||||
parent_channel_id = str(message.channel.parent.id)
|
||||
|
||||
await self._enqueue_raw(
|
||||
RawIncoming(
|
||||
sender_id=user_id,
|
||||
chat_id=parent_channel_id,
|
||||
text=text,
|
||||
media_files=media_paths,
|
||||
content_annotations=annotations,
|
||||
timestamp=message.created_at or datetime.now(),
|
||||
message_id=str(message.id),
|
||||
metadata={"chat_id": parent_channel_id, "thread_id": thread_id},
|
||||
is_group=not is_dm,
|
||||
was_mentioned=was_mentioned,
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Discord bot token validation."""
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def validate_discord_token(
|
||||
token: str, proxy: str | None = None
|
||||
) -> tuple[bool, str]:
|
||||
"""Validate a Discord bot token via the REST API.
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, message).
|
||||
"""
|
||||
if not token:
|
||||
return False, "No token provided"
|
||||
|
||||
try:
|
||||
import httpx
|
||||
except ImportError:
|
||||
return False, "httpx not installed"
|
||||
|
||||
url = "https://discord.com/api/v10/users/@me"
|
||||
headers = {"Authorization": f"Bot {token}"}
|
||||
try:
|
||||
async with httpx.AsyncClient(proxy=proxy) as client:
|
||||
resp = await client.get(url, headers=headers, timeout=10)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
username = data.get("username", "unknown")
|
||||
return True, f"Bot: {username}"
|
||||
return False, "Invalid token"
|
||||
except Exception as e:
|
||||
return False, f"Error: {e}"
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Discord channel server.
|
||||
|
||||
Standalone script to run the Discord channel with CLI options.
|
||||
|
||||
Usage:
|
||||
python -m EvoScientist.channels.discord.serve --bot-token TOKEN [OPTIONS]
|
||||
|
||||
Examples:
|
||||
# Allow all senders (default)
|
||||
python -m EvoScientist.channels.discord.serve --bot-token TOKEN
|
||||
|
||||
# Only allow specific senders and channels
|
||||
python -m EvoScientist.channels.discord.serve --bot-token TOKEN --allow 123 --allow-channel 456
|
||||
|
||||
# With proxy, agent and thinking
|
||||
python -m EvoScientist.channels.discord.serve --bot-token TOKEN --proxy http://proxy:8080 --agent --thinking
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
from ...logging_config import configure_logging_from_settings
|
||||
from ..bus import MessageBus
|
||||
from ..standalone import run_standalone
|
||||
from .channel import DiscordChannel, DiscordConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Discord channel server",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--bot-token",
|
||||
required=True,
|
||||
help="Discord bot token",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow",
|
||||
action="append",
|
||||
dest="allowed_senders",
|
||||
help="Allowed sender (Discord user ID). Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow-channel",
|
||||
action="append",
|
||||
dest="allowed_channels",
|
||||
help="Allowed channel ID. Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--proxy",
|
||||
help="HTTP proxy URL for Discord API requests",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--agent",
|
||||
action="store_true",
|
||||
help="Use EvoScientist agent as handler (default: echo)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--thinking",
|
||||
action="store_true",
|
||||
help="Send thinking content as intermediate messages (requires --agent)",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
"""Entry point."""
|
||||
configure_logging_from_settings(default_level=logging.INFO)
|
||||
args = parse_args()
|
||||
|
||||
config = DiscordConfig(
|
||||
bot_token=args.bot_token,
|
||||
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
|
||||
allowed_channels=set(args.allowed_channels) if args.allowed_channels else None,
|
||||
proxy=args.proxy,
|
||||
)
|
||||
|
||||
send_thinking = args.thinking and args.agent
|
||||
bus = MessageBus()
|
||||
channel = DiscordChannel(config)
|
||||
|
||||
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Email channel for EvoScientist.
|
||||
|
||||
Uses IMAP polling for inbound + SMTP for outbound. Pure Python, no extra deps.
|
||||
|
||||
Usage in config:
|
||||
channel_enabled = "email"
|
||||
email_imap_host = "imap.gmail.com"
|
||||
email_smtp_host = "smtp.gmail.com"
|
||||
...
|
||||
"""
|
||||
|
||||
from ..channel_manager import _parse_csv, register_channel
|
||||
from .channel import EmailChannel, EmailConfig
|
||||
|
||||
__all__ = ["EmailChannel", "EmailConfig"]
|
||||
|
||||
|
||||
def create_from_config(config) -> EmailChannel:
|
||||
allowed = _parse_csv(config.email_allowed_senders)
|
||||
return EmailChannel(
|
||||
EmailConfig(
|
||||
imap_host=config.email_imap_host,
|
||||
imap_port=config.email_imap_port,
|
||||
imap_username=config.email_imap_username,
|
||||
imap_password=config.email_imap_password,
|
||||
imap_mailbox=config.email_imap_mailbox,
|
||||
imap_use_ssl=config.email_imap_use_ssl,
|
||||
smtp_host=config.email_smtp_host,
|
||||
smtp_port=config.email_smtp_port,
|
||||
smtp_username=config.email_smtp_username,
|
||||
smtp_password=config.email_smtp_password,
|
||||
smtp_starttls=config.email_smtp_use_tls,
|
||||
from_address=config.email_from_address,
|
||||
poll_interval=config.email_poll_interval,
|
||||
mark_seen=config.email_mark_seen,
|
||||
max_body_chars=config.email_max_body_chars,
|
||||
subject_prefix=config.email_subject_prefix,
|
||||
allowed_senders=allowed,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
register_channel("email", create_from_config)
|
||||
@@ -0,0 +1,454 @@
|
||||
"""Email channel implementation using IMAP + SMTP."""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import email as email_lib
|
||||
import email.utils
|
||||
import html
|
||||
import imaplib
|
||||
import logging
|
||||
import re
|
||||
import smtplib
|
||||
import ssl
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from email import encoders
|
||||
from email.header import decode_header, make_header
|
||||
from email.message import EmailMessage
|
||||
from email.mime.base import MIMEBase
|
||||
from email.mime.multipart import MIMEMultipart
|
||||
from email.mime.text import MIMEText
|
||||
from email.utils import parseaddr
|
||||
from pathlib import Path
|
||||
|
||||
from ..base import Channel, ChannelError, RawIncoming
|
||||
from ..capabilities import EMAIL as EMAIL_CAPS
|
||||
from ..config import BaseChannelConfig
|
||||
from ..mixins import PollingMixin
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _decode_hdr(raw: str) -> str:
|
||||
try:
|
||||
return str(make_header(decode_header(raw))) if raw else ""
|
||||
except Exception:
|
||||
return raw or ""
|
||||
|
||||
|
||||
def _strip_html(text: str) -> str:
|
||||
text = re.sub(r"<br\s*/?>", "\n", text, flags=re.IGNORECASE)
|
||||
text = re.sub(r"<p[^>]*>", "\n", text, flags=re.IGNORECASE)
|
||||
text = re.sub(r"</p>", "\n", text, flags=re.IGNORECASE)
|
||||
text = re.sub(r"<[^>]+>", "", text)
|
||||
return html.unescape(text).strip()
|
||||
|
||||
|
||||
@dataclass
|
||||
class EmailConfig(BaseChannelConfig):
|
||||
imap_host: str = ""
|
||||
imap_port: int = 993
|
||||
imap_username: str = ""
|
||||
imap_password: str = ""
|
||||
imap_mailbox: str = "INBOX"
|
||||
imap_use_ssl: bool = True
|
||||
smtp_host: str = ""
|
||||
smtp_port: int = 587
|
||||
smtp_username: str = ""
|
||||
smtp_password: str = ""
|
||||
smtp_starttls: bool = (
|
||||
True # True=STARTTLS (port 587), False=implicit SSL (port 465)
|
||||
)
|
||||
from_address: str = ""
|
||||
poll_interval: int = 30
|
||||
mark_seen: bool = True
|
||||
max_body_chars: int = 12000
|
||||
subject_prefix: str = "Re: "
|
||||
text_chunk_limit: int = 4096
|
||||
|
||||
|
||||
class EmailChannel(Channel, PollingMixin):
|
||||
"""Email channel using IMAP polling + SMTP."""
|
||||
|
||||
name = "email"
|
||||
|
||||
capabilities = EMAIL_CAPS
|
||||
_non_retryable_patterns = ("auth", "login", "credential")
|
||||
|
||||
def __init__(self, config: EmailConfig):
|
||||
super().__init__(config)
|
||||
self._imap: imaplib.IMAP4_SSL | imaplib.IMAP4 | None = None
|
||||
|
||||
async def start(self) -> None:
|
||||
cfg = self.config
|
||||
if not cfg.imap_host or not cfg.imap_username:
|
||||
raise ChannelError("Email imap_host and imap_username are required")
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(None, self._connect_imap)
|
||||
self._running = True
|
||||
logger.info(
|
||||
f"Email channel started (IMAP: {cfg.imap_host}, poll {cfg.poll_interval}s)"
|
||||
)
|
||||
await self._start_polling()
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
await self._stop_polling()
|
||||
if self._imap:
|
||||
try:
|
||||
self._imap.close()
|
||||
self._imap.logout()
|
||||
except Exception:
|
||||
pass
|
||||
self._imap = None
|
||||
logger.info("Email channel stopped")
|
||||
|
||||
def _connect_imap(self) -> None:
|
||||
cfg = self.config
|
||||
try:
|
||||
if cfg.imap_use_ssl:
|
||||
self._imap = imaplib.IMAP4_SSL(
|
||||
cfg.imap_host,
|
||||
cfg.imap_port,
|
||||
ssl_context=ssl.create_default_context(),
|
||||
)
|
||||
else:
|
||||
self._imap = imaplib.IMAP4(cfg.imap_host, cfg.imap_port)
|
||||
self._imap.login(cfg.imap_username, cfg.imap_password)
|
||||
self._imap.select(cfg.imap_mailbox)
|
||||
except Exception as e:
|
||||
raise ChannelError(f"IMAP failed: {e}") from e
|
||||
|
||||
def _reconnect_imap(self) -> None:
|
||||
try:
|
||||
if self._imap:
|
||||
self._imap.noop()
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
self._connect_imap()
|
||||
|
||||
async def _poll_once(self) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
messages = await loop.run_in_executor(None, self._fetch_unseen)
|
||||
for m in messages:
|
||||
await self._process_email(m)
|
||||
|
||||
def _fetch_unseen(self) -> list[dict]:
|
||||
self._reconnect_imap()
|
||||
results = []
|
||||
try:
|
||||
st, data = self._imap.search(None, "UNSEEN")
|
||||
if st != "OK":
|
||||
return []
|
||||
for mid in data[0].split()[-20:]:
|
||||
st, msg_data = self._imap.fetch(mid, "(RFC822)")
|
||||
if st != "OK":
|
||||
continue
|
||||
msg = email_lib.message_from_bytes(msg_data[0][1])
|
||||
from_name, from_addr = parseaddr(msg.get("From", ""))
|
||||
body = self._extract_body(msg)
|
||||
if len(body) > self.config.max_body_chars:
|
||||
body = body[: self.config.max_body_chars] + "\n[...truncated]"
|
||||
# Extract attachments and inline images
|
||||
attachments = []
|
||||
if msg.is_multipart():
|
||||
for part in msg.walk():
|
||||
content_disp = part.get("Content-Disposition") or ""
|
||||
content_type = part.get_content_type() or ""
|
||||
is_attachment = "attachment" in content_disp.lower()
|
||||
is_inline_image = (
|
||||
"inline" in content_disp.lower()
|
||||
and content_type.startswith("image/")
|
||||
)
|
||||
# Also detect non-text parts with a filename but no
|
||||
# Content-Disposition header (common for PDFs, docs,
|
||||
# etc. sent by some email clients).
|
||||
is_named_file = (
|
||||
not is_attachment
|
||||
and not is_inline_image
|
||||
and part.get_filename()
|
||||
and not content_type.startswith("multipart/")
|
||||
and not content_type.startswith("text/")
|
||||
)
|
||||
if is_attachment or is_inline_image or is_named_file:
|
||||
filename = part.get_filename() or "attachment"
|
||||
filename = _decode_hdr(filename)
|
||||
payload_data = part.get_payload(decode=True)
|
||||
if payload_data:
|
||||
from ..base import MAX_ATTACHMENT_BYTES, MEDIA_DIR
|
||||
|
||||
if len(payload_data) > MAX_ATTACHMENT_BYTES:
|
||||
attachments.append(
|
||||
{
|
||||
"annotation": f"[attachment: {filename} - too large ({len(payload_data)} bytes)]"
|
||||
}
|
||||
)
|
||||
else:
|
||||
MEDIA_DIR.mkdir(parents=True, exist_ok=True)
|
||||
local_path = (
|
||||
MEDIA_DIR / f"email_{mid.decode()}_{filename}"
|
||||
)
|
||||
local_path.write_bytes(payload_data)
|
||||
label = (
|
||||
"inline-image"
|
||||
if is_inline_image
|
||||
else "attachment"
|
||||
)
|
||||
attachments.append(
|
||||
{
|
||||
"path": str(local_path),
|
||||
"annotation": f"[{label}: {local_path}]",
|
||||
}
|
||||
)
|
||||
if self.config.mark_seen:
|
||||
self._imap.store(mid, "+FLAGS", "\\Seen")
|
||||
results.append(
|
||||
{
|
||||
"from_addr": from_addr,
|
||||
"from_name": _decode_hdr(from_name),
|
||||
"subject": _decode_hdr(msg.get("Subject", "")),
|
||||
"body": body,
|
||||
"message_id": msg.get("Message-ID", ""),
|
||||
"date": msg.get("Date", ""),
|
||||
"references": msg.get("References", ""),
|
||||
"attachments": attachments,
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"IMAP fetch: {e}")
|
||||
return results
|
||||
|
||||
def _extract_body(self, msg) -> str:
|
||||
if msg.is_multipart():
|
||||
for part in msg.walk():
|
||||
ct = part.get_content_type()
|
||||
if ct == "text/plain":
|
||||
return self._decode_payload(part)
|
||||
for part in msg.walk():
|
||||
if part.get_content_type() == "text/html":
|
||||
return _strip_html(self._decode_payload(part))
|
||||
return "[no text content]"
|
||||
text = self._decode_payload(msg)
|
||||
return _strip_html(text) if msg.get_content_type() == "text/html" else text
|
||||
|
||||
@staticmethod
|
||||
def _decode_payload(part) -> str:
|
||||
payload = part.get_payload(decode=True)
|
||||
if not payload:
|
||||
return ""
|
||||
charset = part.get_content_charset() or "utf-8"
|
||||
return payload.decode(charset, errors="replace")
|
||||
|
||||
async def _process_email(self, m: dict) -> None:
|
||||
subject = m["subject"]
|
||||
text = f"[邮件] 主题: {subject}\n\n{m['body']}" if subject else m["body"]
|
||||
try:
|
||||
ts = email_lib.utils.parsedate_to_datetime(m["date"])
|
||||
except Exception:
|
||||
ts = datetime.now()
|
||||
# Process attachments
|
||||
media_paths: list[str] = []
|
||||
annotations: list[str] = []
|
||||
for att in m.get("attachments", []):
|
||||
if att.get("path"):
|
||||
media_paths.append(att["path"])
|
||||
if att.get("annotation"):
|
||||
annotations.append(att["annotation"])
|
||||
await self._enqueue_raw(
|
||||
RawIncoming(
|
||||
sender_id=m["from_addr"],
|
||||
chat_id=m["from_addr"],
|
||||
text=text,
|
||||
timestamp=ts,
|
||||
message_id=m["message_id"],
|
||||
media_files=media_paths,
|
||||
content_annotations=annotations,
|
||||
metadata={
|
||||
"chat_id": m["from_addr"],
|
||||
"subject": subject,
|
||||
"original_message_id": m["message_id"],
|
||||
"references": m["references"],
|
||||
"backend": "email",
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
# ── Send ──────────────────────────────────────────────────────
|
||||
|
||||
def _is_ready(self) -> bool:
|
||||
return bool(self.config.smtp_host)
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _smtp_connect(self):
|
||||
"""Open an SMTP connection as a context manager.
|
||||
|
||||
Ensures the connection is closed even if login or send raises.
|
||||
Uses STARTTLS (port 587) when smtp_starttls=True, otherwise
|
||||
implicit SSL (port 465).
|
||||
"""
|
||||
cfg = self.config
|
||||
srv = None
|
||||
try:
|
||||
if cfg.smtp_starttls:
|
||||
srv = smtplib.SMTP(cfg.smtp_host, cfg.smtp_port, timeout=30)
|
||||
srv.starttls()
|
||||
else:
|
||||
srv = smtplib.SMTP_SSL(
|
||||
cfg.smtp_host,
|
||||
cfg.smtp_port,
|
||||
context=ssl.create_default_context(),
|
||||
timeout=30,
|
||||
)
|
||||
srv.login(cfg.smtp_username, cfg.smtp_password)
|
||||
yield srv
|
||||
finally:
|
||||
if srv is not None:
|
||||
try:
|
||||
srv.quit()
|
||||
except Exception:
|
||||
try:
|
||||
srv.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
|
||||
loop = asyncio.get_running_loop()
|
||||
try:
|
||||
await loop.run_in_executor(
|
||||
None,
|
||||
self._smtp_send_html,
|
||||
chat_id,
|
||||
formatted_text,
|
||||
raw_text,
|
||||
metadata or {},
|
||||
)
|
||||
except Exception as e:
|
||||
err_str = str(e).lower()
|
||||
# Only fall back to plain text for format-related errors, not server rejections
|
||||
if any(
|
||||
code in err_str for code in ("550", "553", "554", "auth", "rejected")
|
||||
):
|
||||
raise
|
||||
logger.warning(f"HTML email failed ({e}), falling back to plain text")
|
||||
await loop.run_in_executor(
|
||||
None,
|
||||
self._smtp_send,
|
||||
chat_id,
|
||||
raw_text,
|
||||
metadata or {},
|
||||
)
|
||||
|
||||
def _smtp_send(self, to: str, content: str, meta: dict) -> None:
|
||||
cfg = self.config
|
||||
from_addr = cfg.from_address or cfg.smtp_username
|
||||
logger.debug(f"SMTP plain send: from={from_addr} to={to}")
|
||||
msg = EmailMessage()
|
||||
orig_subj = meta.get("subject", "")
|
||||
msg["Subject"] = (
|
||||
f"{cfg.subject_prefix}{orig_subj}"
|
||||
if orig_subj and not orig_subj.lower().startswith("re:")
|
||||
else (orig_subj or "EvoScientist Reply")
|
||||
)
|
||||
msg["From"] = from_addr
|
||||
msg["To"] = to
|
||||
orig_id = meta.get("original_message_id", "")
|
||||
if orig_id:
|
||||
msg["In-Reply-To"] = orig_id
|
||||
msg["References"] = f"{meta.get('references', '')} {orig_id}".strip()
|
||||
msg.set_content(content)
|
||||
try:
|
||||
with self._smtp_connect() as srv:
|
||||
srv.sendmail(from_addr, [to], msg.as_string())
|
||||
except Exception as e:
|
||||
logger.error(f"SMTP send failed: from={from_addr} to={to}")
|
||||
raise RuntimeError("SMTP send failed") from e
|
||||
|
||||
def _smtp_send_html(
|
||||
self, to: str, html_content: str, plain_content: str, meta: dict
|
||||
) -> None:
|
||||
"""Send an email with both HTML and plain-text parts."""
|
||||
cfg = self.config
|
||||
from_addr = cfg.from_address or cfg.smtp_username
|
||||
logger.debug(f"SMTP HTML send: from={from_addr} to={to}")
|
||||
msg = MIMEMultipart("alternative")
|
||||
orig_subj = meta.get("subject", "")
|
||||
msg["Subject"] = (
|
||||
f"{cfg.subject_prefix}{orig_subj}"
|
||||
if orig_subj and not orig_subj.lower().startswith("re:")
|
||||
else (orig_subj or "EvoScientist Reply")
|
||||
)
|
||||
msg["From"] = from_addr
|
||||
msg["To"] = to
|
||||
orig_id = meta.get("original_message_id", "")
|
||||
if orig_id:
|
||||
msg["In-Reply-To"] = orig_id
|
||||
msg["References"] = f"{meta.get('references', '')} {orig_id}".strip()
|
||||
msg.attach(MIMEText(plain_content, "plain", "utf-8"))
|
||||
msg.attach(MIMEText(html_content, "html", "utf-8"))
|
||||
try:
|
||||
with self._smtp_connect() as srv:
|
||||
srv.sendmail(from_addr, [to], msg.as_string())
|
||||
except Exception as e:
|
||||
logger.error(f"SMTP HTML send failed: from={from_addr} to={to}")
|
||||
raise RuntimeError("SMTP HTML send failed") from e
|
||||
|
||||
# ── Media send (email attachment) ─────────────────────────────
|
||||
|
||||
async def _send_media_impl(
|
||||
self,
|
||||
recipient: str,
|
||||
file_path: str,
|
||||
caption: str = "",
|
||||
metadata: dict | None = None,
|
||||
) -> bool:
|
||||
"""Send a file as an email attachment via SMTP."""
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(
|
||||
None,
|
||||
self._smtp_send_attachment,
|
||||
recipient,
|
||||
file_path,
|
||||
caption,
|
||||
metadata or {},
|
||||
)
|
||||
return True
|
||||
|
||||
def _smtp_send_attachment(
|
||||
self, to: str, file_path: str, caption: str, meta: dict
|
||||
) -> None:
|
||||
"""Send an email with a file attachment."""
|
||||
cfg = self.config
|
||||
from_addr = cfg.from_address or cfg.smtp_username
|
||||
logger.debug(f"SMTP attachment send: from={from_addr} to={to} file={file_path}")
|
||||
msg = MIMEMultipart()
|
||||
orig_subj = meta.get("subject", "")
|
||||
msg["Subject"] = (
|
||||
f"{cfg.subject_prefix}{orig_subj}"
|
||||
if orig_subj and not orig_subj.lower().startswith("re:")
|
||||
else (orig_subj or "EvoScientist Reply")
|
||||
)
|
||||
msg["From"] = from_addr
|
||||
msg["To"] = to
|
||||
orig_id = meta.get("original_message_id", "")
|
||||
if orig_id:
|
||||
msg["In-Reply-To"] = orig_id
|
||||
msg["References"] = f"{meta.get('references', '')} {orig_id}".strip()
|
||||
|
||||
# Text body
|
||||
if caption:
|
||||
msg.attach(MIMEText(caption, "plain", "utf-8"))
|
||||
|
||||
# Attachment
|
||||
path = Path(file_path)
|
||||
part = MIMEBase("application", "octet-stream")
|
||||
part.set_payload(path.read_bytes())
|
||||
encoders.encode_base64(part)
|
||||
part.add_header("Content-Disposition", f"attachment; filename={path.name}")
|
||||
msg.attach(part)
|
||||
|
||||
try:
|
||||
with self._smtp_connect() as srv:
|
||||
srv.sendmail(from_addr, [to], msg.as_string())
|
||||
except Exception as e:
|
||||
logger.error(f"SMTP attachment send failed: from={from_addr} to={to}")
|
||||
raise RuntimeError("SMTP attachment send failed") from e
|
||||
@@ -0,0 +1,92 @@
|
||||
"""Email credential validation."""
|
||||
|
||||
import imaplib
|
||||
import logging
|
||||
import smtplib
|
||||
import ssl
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def validate_email_imap(
|
||||
host: str,
|
||||
port: int,
|
||||
username: str,
|
||||
password: str,
|
||||
use_ssl: bool = True,
|
||||
) -> tuple[bool, str]:
|
||||
"""Validate IMAP credentials.
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, message).
|
||||
"""
|
||||
if not host or not username or not password:
|
||||
return False, "host, username, and password are required"
|
||||
|
||||
import asyncio
|
||||
|
||||
loop = asyncio.get_event_loop()
|
||||
|
||||
def _check():
|
||||
try:
|
||||
if use_ssl:
|
||||
ctx = ssl.create_default_context()
|
||||
conn = imaplib.IMAP4_SSL(host, port, ssl_context=ctx)
|
||||
else:
|
||||
conn = imaplib.IMAP4(host, port)
|
||||
conn.login(username, password)
|
||||
conn.logout()
|
||||
return True, "IMAP credentials valid"
|
||||
except imaplib.IMAP4.error as e:
|
||||
return False, f"IMAP auth failed: {e}"
|
||||
except Exception as e:
|
||||
return False, f"IMAP error: {e}"
|
||||
|
||||
return await loop.run_in_executor(None, _check)
|
||||
|
||||
|
||||
async def validate_email_smtp(
|
||||
host: str,
|
||||
port: int,
|
||||
username: str,
|
||||
password: str,
|
||||
use_tls: bool = True,
|
||||
) -> tuple[bool, str]:
|
||||
"""Validate SMTP credentials.
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, message).
|
||||
"""
|
||||
if not host or not username or not password:
|
||||
return False, "host, username, and password are required"
|
||||
|
||||
import asyncio
|
||||
|
||||
loop = asyncio.get_event_loop()
|
||||
|
||||
def _check():
|
||||
server = None
|
||||
try:
|
||||
if use_tls:
|
||||
server = smtplib.SMTP(host, port, timeout=10)
|
||||
server.starttls()
|
||||
else:
|
||||
ctx = ssl.create_default_context()
|
||||
server = smtplib.SMTP_SSL(host, port, context=ctx, timeout=10)
|
||||
server.login(username, password)
|
||||
return True, "SMTP credentials valid"
|
||||
except smtplib.SMTPAuthenticationError:
|
||||
return False, "SMTP auth failed"
|
||||
except Exception as e:
|
||||
return False, f"SMTP error: {e}"
|
||||
finally:
|
||||
if server is not None:
|
||||
try:
|
||||
server.quit()
|
||||
except Exception:
|
||||
try:
|
||||
server.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return await loop.run_in_executor(None, _check)
|
||||
@@ -0,0 +1,121 @@
|
||||
"""Email channel server.
|
||||
|
||||
Standalone script to run the Email channel with CLI options.
|
||||
|
||||
Usage:
|
||||
python -m EvoScientist.channels.email.serve --imap-host HOST --imap-username USER --imap-password PASS --smtp-host HOST --smtp-username USER --smtp-password PASS --from-address ADDR [OPTIONS]
|
||||
|
||||
Examples:
|
||||
# Basic usage
|
||||
python -m EvoScientist.channels.email.serve --imap-host imap.gmail.com --imap-username bot@gmail.com --imap-password PASS --smtp-host smtp.gmail.com --smtp-username bot@gmail.com --smtp-password PASS --from-address bot@gmail.com
|
||||
|
||||
# With allowed senders and custom poll interval
|
||||
python -m EvoScientist.channels.email.serve --imap-host imap.gmail.com --imap-username bot@gmail.com --imap-password PASS --smtp-host smtp.gmail.com --smtp-username bot@gmail.com --smtp-password PASS --from-address bot@gmail.com --allow user@example.com --poll-interval 60
|
||||
|
||||
# With agent and thinking
|
||||
python -m EvoScientist.channels.email.serve --imap-host imap.gmail.com --imap-username bot@gmail.com --imap-password PASS --smtp-host smtp.gmail.com --smtp-username bot@gmail.com --smtp-password PASS --from-address bot@gmail.com --agent --thinking
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
from ...logging_config import configure_logging_from_settings
|
||||
from ..bus import MessageBus
|
||||
from ..standalone import run_standalone
|
||||
from .channel import EmailChannel, EmailConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Email channel server",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--imap-host",
|
||||
required=True,
|
||||
help="IMAP server hostname",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--imap-username",
|
||||
required=True,
|
||||
help="IMAP username",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--imap-password",
|
||||
required=True,
|
||||
help="IMAP password",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--smtp-host",
|
||||
required=True,
|
||||
help="SMTP server hostname",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--smtp-username",
|
||||
required=True,
|
||||
help="SMTP username",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--smtp-password",
|
||||
required=True,
|
||||
help="SMTP password",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--from-address",
|
||||
required=True,
|
||||
help="From email address for outgoing messages",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow",
|
||||
action="append",
|
||||
dest="allowed_senders",
|
||||
help="Allowed sender (email address). Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--poll-interval",
|
||||
type=int,
|
||||
default=30,
|
||||
help="IMAP poll interval in seconds (default: 30)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--agent",
|
||||
action="store_true",
|
||||
help="Use EvoScientist agent as handler (default: echo)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--thinking",
|
||||
action="store_true",
|
||||
help="Send thinking content as intermediate messages (requires --agent)",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
"""Entry point."""
|
||||
configure_logging_from_settings(default_level=logging.INFO)
|
||||
args = parse_args()
|
||||
|
||||
config = EmailConfig(
|
||||
imap_host=args.imap_host,
|
||||
imap_username=args.imap_username,
|
||||
imap_password=args.imap_password,
|
||||
smtp_host=args.smtp_host,
|
||||
smtp_username=args.smtp_username,
|
||||
smtp_password=args.smtp_password,
|
||||
from_address=args.from_address,
|
||||
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
|
||||
poll_interval=args.poll_interval,
|
||||
)
|
||||
|
||||
send_thinking = args.thinking and args.agent
|
||||
bus = MessageBus()
|
||||
channel = EmailChannel(config)
|
||||
|
||||
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,25 @@
|
||||
from ..channel_manager import _parse_csv, register_channel
|
||||
from .channel import FeishuChannel, FeishuConfig
|
||||
|
||||
__all__ = ["FeishuChannel", "FeishuConfig"]
|
||||
|
||||
|
||||
def create_from_config(config) -> FeishuChannel:
|
||||
allowed = _parse_csv(config.feishu_allowed_senders)
|
||||
proxy = config.feishu_proxy or None
|
||||
return FeishuChannel(
|
||||
FeishuConfig(
|
||||
app_id=config.feishu_app_id,
|
||||
app_secret=config.feishu_app_secret,
|
||||
verification_token=config.feishu_verification_token,
|
||||
encrypt_key=config.feishu_encrypt_key,
|
||||
webhook_port=config.feishu_webhook_port,
|
||||
allowed_senders=allowed,
|
||||
feishu_domain=config.feishu_domain,
|
||||
proxy=proxy,
|
||||
subscription_mode=getattr(config, "feishu_subscription_mode", "webhook"),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
register_channel("feishu", create_from_config)
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Feishu (飞书/Lark) app credential validation."""
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def validate_feishu_credentials(
|
||||
app_id: str,
|
||||
app_secret: str,
|
||||
domain: str = "https://open.feishu.cn",
|
||||
) -> tuple[bool, str]:
|
||||
"""Validate Feishu app credentials by requesting a tenant_access_token.
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, message).
|
||||
"""
|
||||
if not app_id:
|
||||
return False, "No app_id provided"
|
||||
if not app_secret:
|
||||
return False, "No app_secret provided"
|
||||
|
||||
try:
|
||||
import httpx
|
||||
except ImportError:
|
||||
return False, "httpx not installed"
|
||||
|
||||
url = f"{domain}/open-apis/auth/v3/tenant_access_token/internal"
|
||||
body = {"app_id": app_id, "app_secret": app_secret}
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(url, json=body, timeout=10)
|
||||
data = resp.json()
|
||||
if data.get("code") == 0:
|
||||
return True, f"App: {app_id}"
|
||||
msg = data.get("msg", "unknown error")
|
||||
return False, f"Auth failed: {msg}"
|
||||
except Exception as e:
|
||||
return False, f"Error: {e}"
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Feishu (飞书/Lark) channel server.
|
||||
|
||||
Standalone script to run the Feishu channel with CLI options.
|
||||
|
||||
Usage:
|
||||
python -m EvoScientist.channels.feishu.serve --app-id ID --app-secret SECRET [OPTIONS]
|
||||
|
||||
Examples:
|
||||
# Basic setup
|
||||
python -m EvoScientist.channels.feishu.serve --app-id ID --app-secret SECRET
|
||||
|
||||
# With verification token and custom port
|
||||
python -m EvoScientist.channels.feishu.serve --app-id ID --app-secret SECRET \\
|
||||
--verification-token TOKEN --webhook-port 9000
|
||||
|
||||
# With agent and thinking
|
||||
python -m EvoScientist.channels.feishu.serve --app-id ID --app-secret SECRET --agent --thinking
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
from ...logging_config import configure_logging_from_settings
|
||||
from ..bus import MessageBus
|
||||
from ..standalone import run_standalone
|
||||
from .channel import FeishuChannel, FeishuConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Feishu (飞书/Lark) channel server",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--app-id",
|
||||
required=True,
|
||||
help="Feishu App ID",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--app-secret",
|
||||
required=True,
|
||||
help="Feishu App Secret",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verification-token",
|
||||
default="",
|
||||
help="Feishu event verification token",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--encrypt-key",
|
||||
default="",
|
||||
help="Feishu event encrypt key",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--webhook-port",
|
||||
type=int,
|
||||
default=9000,
|
||||
help="Port for webhook HTTP server (default: 9000)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--domain",
|
||||
default="https://open.feishu.cn",
|
||||
help="Feishu API domain (use https://open.larksuite.com for Lark)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow",
|
||||
action="append",
|
||||
dest="allowed_senders",
|
||||
help="Allowed sender (Feishu open_id). Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mode",
|
||||
choices=["webhook", "websocket"],
|
||||
default="webhook",
|
||||
help="Subscription mode: webhook (default) or websocket (long connection, no public IP needed)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--agent",
|
||||
action="store_true",
|
||||
help="Use EvoScientist agent as handler (default: echo)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--thinking",
|
||||
action="store_true",
|
||||
help="Send thinking content as intermediate messages (requires --agent)",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
"""Entry point."""
|
||||
configure_logging_from_settings(default_level=logging.INFO)
|
||||
args = parse_args()
|
||||
|
||||
config = FeishuConfig(
|
||||
app_id=args.app_id,
|
||||
app_secret=args.app_secret,
|
||||
verification_token=args.verification_token,
|
||||
encrypt_key=args.encrypt_key,
|
||||
webhook_port=args.webhook_port,
|
||||
feishu_domain=args.domain,
|
||||
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
|
||||
subscription_mode=args.mode,
|
||||
)
|
||||
|
||||
send_thinking = args.thinking and args.agent
|
||||
bus = MessageBus()
|
||||
channel = FeishuChannel(config)
|
||||
|
||||
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,295 @@
|
||||
"""Unified formatting pipeline for all channels.
|
||||
|
||||
Internal representation is Markdown. This module converts Markdown to
|
||||
each platform's native format: HTML, Slack mrkdwn, Discord Markdown,
|
||||
or plain text.
|
||||
|
||||
Channels no longer need per-file format functions — they just declare
|
||||
``capabilities.format_type`` and the base class auto-configures a
|
||||
``UnifiedFormatter`` instance.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Callable
|
||||
from typing import ClassVar
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Markdown conversion engine (formerly markdown_utils.py)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
_PLACEHOLDER_PREFIX = "\x00BLOCK"
|
||||
_INLINE_PREFIX = "\x00INLINE"
|
||||
|
||||
# A formatting rule: (regex_pattern, replacement)
|
||||
InlineRule = tuple[str, str]
|
||||
|
||||
|
||||
def convert_markdown(
|
||||
text: str,
|
||||
*,
|
||||
code_block_formatter: Callable[[str, str], str],
|
||||
inline_code_formatter: Callable[[str], str],
|
||||
inline_rules: list[InlineRule],
|
||||
escape_fn: Callable[[str], str] | None = None,
|
||||
) -> str:
|
||||
"""Convert Markdown to a channel-specific format.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
text:
|
||||
Input Markdown text.
|
||||
code_block_formatter:
|
||||
``(language, code) -> str`` — format a fenced code block.
|
||||
inline_code_formatter:
|
||||
``(code) -> str`` — format an inline code span.
|
||||
inline_rules:
|
||||
List of ``(pattern, replacement)`` pairs applied in order to the
|
||||
remaining text (after code extraction and optional escaping).
|
||||
escape_fn:
|
||||
Optional function applied to the non-code text *before* inline
|
||||
rules. Useful for HTML-escaping (Telegram) or other channel-
|
||||
specific character escaping.
|
||||
|
||||
Returns
|
||||
-------
|
||||
str
|
||||
The converted text.
|
||||
"""
|
||||
# 1. Extract and protect fenced code blocks (```...```)
|
||||
code_blocks: list[str] = []
|
||||
|
||||
def _save_code_block(m: re.Match) -> str:
|
||||
lang = m.group(1) or ""
|
||||
code = m.group(2)
|
||||
formatted = code_block_formatter(lang, code)
|
||||
idx = len(code_blocks)
|
||||
code_blocks.append(formatted)
|
||||
return f"{_PLACEHOLDER_PREFIX}{idx}\x00"
|
||||
|
||||
text = re.sub(r"```(\w*)\n?(.*?)```", _save_code_block, text, flags=re.DOTALL)
|
||||
|
||||
# 2. Extract and protect inline code (`...`)
|
||||
inline_codes: list[str] = []
|
||||
|
||||
def _save_inline(m: re.Match) -> str:
|
||||
code = m.group(1)
|
||||
formatted = inline_code_formatter(code)
|
||||
idx = len(inline_codes)
|
||||
inline_codes.append(formatted)
|
||||
return f"{_INLINE_PREFIX}{idx}\x00"
|
||||
|
||||
text = re.sub(r"`([^`]+)`", _save_inline, text)
|
||||
|
||||
# 3. Optional escaping of remaining text
|
||||
if escape_fn is not None:
|
||||
text = escape_fn(text)
|
||||
|
||||
# 4. Apply inline formatting rules
|
||||
for pattern, replacement in inline_rules:
|
||||
text = re.sub(pattern, replacement, text, flags=re.MULTILINE)
|
||||
|
||||
# 5. Restore code blocks and inline code
|
||||
for idx, html in enumerate(code_blocks):
|
||||
text = text.replace(f"{_PLACEHOLDER_PREFIX}{idx}\x00", html)
|
||||
for idx, code in enumerate(inline_codes):
|
||||
text = text.replace(f"{_INLINE_PREFIX}{idx}\x00", code)
|
||||
|
||||
return text
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Shared helpers
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def _escape_html(text: str) -> str:
|
||||
return text.replace("&", "&").replace("<", "<").replace(">", ">")
|
||||
|
||||
|
||||
def _noop_escape(text: str) -> str:
|
||||
return text
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# HTML profile (Telegram, Email, Teams)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def _html_code_block(lang: str, code: str) -> str:
|
||||
escaped = _escape_html(code)
|
||||
if lang:
|
||||
return f'<pre><code class="language-{lang}">{escaped}</code></pre>'
|
||||
return f"<pre><code>{escaped}</code></pre>"
|
||||
|
||||
|
||||
def _html_inline_code(code: str) -> str:
|
||||
return f"<code>{_escape_html(code)}</code>"
|
||||
|
||||
|
||||
_HTML_INLINE_RULES: list[InlineRule] = [
|
||||
# Headings → bold
|
||||
(r"^#{1,6}\s+(.+)$", r"<b>\1</b>"),
|
||||
# Blockquote markers (already escaped to >)
|
||||
(r"^>\s?", ""),
|
||||
# Links [text](url) → <a>
|
||||
(r"\[([^\]]+)\]\(([^)]+)\)", r'<a href="\2">\1</a>'),
|
||||
# Bold **text** → <b>
|
||||
(r"\*\*(.+?)\*\*", r"<b>\1</b>"),
|
||||
# Italic _text_ → <i>
|
||||
(r"(?<!\w)_([^_]+?)_(?!\w)", r"<i>\1</i>"),
|
||||
# Strikethrough ~~text~~ → <s>
|
||||
(r"~~(.+?)~~", r"<s>\1</s>"),
|
||||
# List items
|
||||
(r"^[\-\*]\s+", "• "),
|
||||
]
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Slack mrkdwn profile
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def _slack_code_block(lang: str, code: str) -> str:
|
||||
return f"```\n{code}```"
|
||||
|
||||
|
||||
def _slack_inline_code(code: str) -> str:
|
||||
return f"`{code}`"
|
||||
|
||||
|
||||
_SLACK_INLINE_RULES: list[InlineRule] = [
|
||||
(r"^#{1,6}\s+(.+)$", r"*\1*"),
|
||||
(r"\[([^\]]+)\]\(([^)]+)\)", r"<\2|\1>"),
|
||||
(r"\*\*(.+?)\*\*", r"*\1*"),
|
||||
(r"~~(.+?)~~", r"~\1~"),
|
||||
(r"^[\-\*]\s+", "• "),
|
||||
]
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Discord profile (mostly passthrough, headings → bold)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def _discord_code_block(lang: str, code: str) -> str:
|
||||
return f"```{lang}\n{code}```"
|
||||
|
||||
|
||||
def _discord_inline_code(code: str) -> str:
|
||||
return f"`{code}`"
|
||||
|
||||
|
||||
_DISCORD_INLINE_RULES: list[InlineRule] = [
|
||||
(r"^#{1,6}\s+(.+)$", r"**\1**"),
|
||||
]
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Plain text profile (strip all formatting)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def _plain_code_block(lang: str, code: str) -> str:
|
||||
return code
|
||||
|
||||
|
||||
def _plain_inline_code(code: str) -> str:
|
||||
return code
|
||||
|
||||
|
||||
_PLAIN_INLINE_RULES: list[InlineRule] = [
|
||||
(r"^#{1,6}\s+", ""),
|
||||
(r"\[([^\]]+)\]\(([^)]+)\)", r"\1 (\2)"),
|
||||
(r"\*\*(.+?)\*\*", r"\1"),
|
||||
(r"(?<!\w)_([^_]+?)_(?!\w)", r"\1"),
|
||||
(r"~~(.+?)~~", r"\1"),
|
||||
(r"^[\-\*]\s+", "• "),
|
||||
]
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Markdown passthrough profile (Feishu, DingTalk, WeCom)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def _md_code_block(lang: str, code: str) -> str:
|
||||
return f"```{lang}\n{code}```"
|
||||
|
||||
|
||||
def _md_inline_code(code: str) -> str:
|
||||
return f"`{code}`"
|
||||
|
||||
|
||||
_MD_INLINE_RULES: list[InlineRule] = [] # passthrough — already Markdown
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Unified Formatter
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class UnifiedFormatter:
|
||||
"""Converts internal Markdown to a target platform format.
|
||||
|
||||
Instantiated once per channel based on its ``capabilities.format_type``.
|
||||
"""
|
||||
|
||||
_PROFILES: ClassVar[dict[str, dict]] = {
|
||||
"html": {
|
||||
"code_block_formatter": _html_code_block,
|
||||
"inline_code_formatter": _html_inline_code,
|
||||
"inline_rules": _HTML_INLINE_RULES,
|
||||
"escape_fn": _escape_html,
|
||||
},
|
||||
"slack_mrkdwn": {
|
||||
"code_block_formatter": _slack_code_block,
|
||||
"inline_code_formatter": _slack_inline_code,
|
||||
"inline_rules": _SLACK_INLINE_RULES,
|
||||
"escape_fn": None,
|
||||
},
|
||||
"discord": {
|
||||
"code_block_formatter": _discord_code_block,
|
||||
"inline_code_formatter": _discord_inline_code,
|
||||
"inline_rules": _DISCORD_INLINE_RULES,
|
||||
"escape_fn": None,
|
||||
},
|
||||
"markdown": {
|
||||
"code_block_formatter": _md_code_block,
|
||||
"inline_code_formatter": _md_inline_code,
|
||||
"inline_rules": _MD_INLINE_RULES,
|
||||
"escape_fn": None,
|
||||
},
|
||||
"plain": {
|
||||
"code_block_formatter": _plain_code_block,
|
||||
"inline_code_formatter": _plain_inline_code,
|
||||
"inline_rules": _PLAIN_INLINE_RULES,
|
||||
"escape_fn": None,
|
||||
},
|
||||
}
|
||||
|
||||
def __init__(self, format_type: str = "plain") -> None:
|
||||
self._format_type = format_type
|
||||
profile = self._PROFILES.get(format_type)
|
||||
if profile is None:
|
||||
raise ValueError(
|
||||
f"Unknown format_type: {format_type!r}. "
|
||||
f"Available: {list(self._PROFILES.keys())}"
|
||||
)
|
||||
self._profile = profile
|
||||
|
||||
@property
|
||||
def format_type(self) -> str:
|
||||
return self._format_type
|
||||
|
||||
def format(self, text: str) -> str:
|
||||
"""Convert Markdown *text* to the target format."""
|
||||
if not text:
|
||||
return text
|
||||
return convert_markdown(text, **self._profile)
|
||||
|
||||
@classmethod
|
||||
def for_channel(cls, format_type: str) -> UnifiedFormatter:
|
||||
"""Factory: create a formatter for the given format type."""
|
||||
return cls(format_type)
|
||||
@@ -0,0 +1,42 @@
|
||||
"""iMessage channel implementation for EvoScientist.
|
||||
|
||||
Uses imsg CLI via JSON-RPC for real-time message streaming.
|
||||
|
||||
Requirements:
|
||||
- macOS only
|
||||
- imsg CLI: brew install steipete/tap/imsg
|
||||
- Full Disk Access permission
|
||||
- Messages.app logged into iCloud
|
||||
"""
|
||||
|
||||
from ..channel_manager import _parse_csv, register_channel
|
||||
from .channel_rpc import IMessageChannelRpc as IMessageChannel
|
||||
from .channel_rpc import IMessageConfig
|
||||
from .probe import ProbeResult, probe_imessage
|
||||
from .targets import (
|
||||
IMessageService,
|
||||
IMessageTarget,
|
||||
normalize_e164,
|
||||
normalize_handle,
|
||||
parse_target,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"IMessageChannel",
|
||||
"IMessageConfig",
|
||||
"IMessageService",
|
||||
"IMessageTarget",
|
||||
"ProbeResult",
|
||||
"normalize_e164",
|
||||
"normalize_handle",
|
||||
"parse_target",
|
||||
"probe_imessage",
|
||||
]
|
||||
|
||||
|
||||
def create_from_config(config) -> IMessageChannel:
|
||||
allowed = _parse_csv(config.imessage_allowed_senders)
|
||||
return IMessageChannel(IMessageConfig(allowed_senders=allowed))
|
||||
|
||||
|
||||
register_channel("imessage", create_from_config)
|
||||
@@ -0,0 +1,416 @@
|
||||
"""iMessage channel using imsg JSON-RPC.
|
||||
|
||||
This is an improved implementation that uses the imsg CLI
|
||||
via JSON-RPC, similar to OpenClaw's approach.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from ..base import Channel, ChannelError, RawIncoming
|
||||
from ..config import BaseChannelConfig
|
||||
from .rpc_client import ImsgRpcClient, RpcNotification
|
||||
from .targets import (
|
||||
ChatGuidTarget,
|
||||
ChatIdentifierTarget,
|
||||
ChatIdTarget,
|
||||
normalize_handle,
|
||||
parse_target,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class _IMessageAllowListMiddleware:
|
||||
"""Custom allow-list middleware for iMessage's rich sender filtering.
|
||||
|
||||
Supports chat_id/chat_guid matching, wildcard, and normalized
|
||||
phone/email matching — logic that the generic AllowListMiddleware
|
||||
does not cover.
|
||||
"""
|
||||
|
||||
def __init__(self, channel: "IMessageChannelRpc"):
|
||||
self._channel = channel
|
||||
|
||||
async def process_inbound(self, raw, context):
|
||||
chat_id = raw.metadata.get("chat_id")
|
||||
chat_guid = raw.metadata.get("chat_guid")
|
||||
if not self._channel._is_sender_allowed(raw.sender_id, chat_id, chat_guid):
|
||||
return None
|
||||
return raw
|
||||
|
||||
|
||||
@dataclass
|
||||
class IMessageConfig(BaseChannelConfig):
|
||||
"""Configuration for iMessage channel."""
|
||||
|
||||
cli_path: str = "imsg"
|
||||
db_path: str | None = None
|
||||
text_chunk_limit: int = 4096
|
||||
service: str = "auto" # imessage, sms, or auto
|
||||
region: str = "US"
|
||||
|
||||
|
||||
class IMessageChannelRpc(Channel):
|
||||
"""iMessage channel using imsg JSON-RPC.
|
||||
|
||||
This implementation uses the imsg CLI via JSON-RPC over stdio,
|
||||
providing real-time message streaming instead of polling.
|
||||
|
||||
Args:
|
||||
config: Channel configuration
|
||||
"""
|
||||
|
||||
name = "imessage"
|
||||
_ready_attrs = ("_client",)
|
||||
|
||||
def __init__(self, config: IMessageConfig | None = None):
|
||||
super().__init__(config or IMessageConfig())
|
||||
self._client: ImsgRpcClient | None = None
|
||||
self._subscription_id: int | None = None
|
||||
self._background_tasks: set[asyncio.Task] = set()
|
||||
|
||||
# ── Pipeline overrides ────────────────────────────────────────
|
||||
|
||||
def _build_inbound_middlewares(self):
|
||||
"""Use iMessage-specific allow-list middleware.
|
||||
|
||||
iMessage doesn't need MentionGating (always sets was_mentioned=True).
|
||||
"""
|
||||
from ..middleware import DedupMiddleware, GroupHistoryMiddleware
|
||||
|
||||
middlewares = []
|
||||
middlewares.append(DedupMiddleware())
|
||||
middlewares.append(_IMessageAllowListMiddleware(self))
|
||||
if self.capabilities.groups:
|
||||
middlewares.append(GroupHistoryMiddleware())
|
||||
return middlewares
|
||||
|
||||
# ── Incoming message handling ─────────────────────────────────
|
||||
|
||||
def _handle_notification(self, notification: RpcNotification) -> None:
|
||||
"""Handle incoming RPC notifications."""
|
||||
if notification.method == "message":
|
||||
_task = asyncio.create_task(self._handle_message(notification.params))
|
||||
self._background_tasks.add(_task)
|
||||
_task.add_done_callback(self._background_tasks.discard)
|
||||
elif notification.method == "error":
|
||||
logger.error(f"imsg error: {notification.params}")
|
||||
|
||||
async def _handle_message(self, params: dict | None) -> None:
|
||||
"""Process incoming message notification."""
|
||||
if not params:
|
||||
return
|
||||
|
||||
message = params.get("message", {})
|
||||
if not message:
|
||||
return
|
||||
|
||||
# Skip messages from self
|
||||
if message.get("is_from_me"):
|
||||
return
|
||||
|
||||
sender = message.get("sender", "").strip()
|
||||
if not sender:
|
||||
return
|
||||
|
||||
text = message.get("text", "").strip()
|
||||
|
||||
# Parse timestamp
|
||||
timestamp = datetime.now()
|
||||
if created_at := message.get("created_at"):
|
||||
try:
|
||||
timestamp = datetime.fromisoformat(created_at)
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
# Build metadata
|
||||
metadata = {
|
||||
"chat_id": message.get("chat_id"),
|
||||
"chat_guid": message.get("chat_guid"),
|
||||
"is_group": message.get("is_group", False),
|
||||
"chat_name": message.get("chat_name"),
|
||||
}
|
||||
|
||||
# Handle attachments if enabled
|
||||
annotations: list[str] = []
|
||||
media_paths: list[str] = []
|
||||
_VOICE_EXTS = {".caf", ".m4a", ".aac", ".ogg", ".opus", ".mp3", ".amr"}
|
||||
if self.config.include_attachments:
|
||||
attachments = message.get("attachments", [])
|
||||
for att in attachments:
|
||||
# imsg CLI provides local file paths for attachments
|
||||
file_path = att if isinstance(att, str) else att.get("path", "")
|
||||
if not file_path:
|
||||
annotations.append("[attachment: missing path]")
|
||||
continue
|
||||
att_path = Path(file_path)
|
||||
is_voice = att_path.suffix.lower() in _VOICE_EXTS
|
||||
media_label = "voice" if is_voice else "attachment"
|
||||
if att_path.exists():
|
||||
fname = att_path.name
|
||||
# Check file size before copying
|
||||
from ..base import MAX_ATTACHMENT_BYTES
|
||||
|
||||
if att_path.stat().st_size > MAX_ATTACHMENT_BYTES:
|
||||
annotations.append(
|
||||
f"[{media_label}: {fname} - too large "
|
||||
f"({att_path.stat().st_size} bytes)]"
|
||||
)
|
||||
else:
|
||||
local = self._media_path(f"imsg_{fname}")
|
||||
try:
|
||||
import shutil
|
||||
|
||||
shutil.copy2(str(att_path), str(local))
|
||||
media_paths.append(str(local))
|
||||
annotations.append(f"[{media_label}: {local}]")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to copy iMessage attachment: {e}")
|
||||
annotations.append(
|
||||
f"[{media_label}: {fname} - copy failed]"
|
||||
)
|
||||
else:
|
||||
annotations.append(f"[{media_label}: {file_path} - not found]")
|
||||
|
||||
if not text and not media_paths and not annotations:
|
||||
return
|
||||
|
||||
is_group = message.get("is_group", False)
|
||||
|
||||
await self._enqueue_raw(
|
||||
RawIncoming(
|
||||
sender_id=sender,
|
||||
chat_id=str(metadata.get("chat_id", sender)),
|
||||
text=text,
|
||||
media_files=media_paths,
|
||||
content_annotations=annotations,
|
||||
timestamp=timestamp,
|
||||
message_id=str(message.get("id", "")),
|
||||
metadata=metadata,
|
||||
is_group=is_group,
|
||||
was_mentioned=True, # iMessage has no mention concept
|
||||
)
|
||||
)
|
||||
|
||||
# ── Sender filtering ──────────────────────────────────────────
|
||||
|
||||
def _is_sender_allowed(
|
||||
self,
|
||||
sender: str,
|
||||
chat_id: int | None = None,
|
||||
chat_guid: str | None = None,
|
||||
) -> bool:
|
||||
"""Check if sender is in allowed list.
|
||||
|
||||
Supports:
|
||||
- Wildcard "*" to allow all
|
||||
- chat_id:123 to match by chat ID
|
||||
- chat_guid:abc to match by chat GUID
|
||||
- Normalized phone/email matching
|
||||
"""
|
||||
if not self.config.allowed_senders:
|
||||
return True
|
||||
|
||||
# Wildcard allows all
|
||||
if "*" in self.config.allowed_senders:
|
||||
return True
|
||||
|
||||
sender_normalized = normalize_handle(sender)
|
||||
|
||||
for entry in self.config.allowed_senders:
|
||||
entry = entry.strip()
|
||||
if not entry:
|
||||
continue
|
||||
|
||||
lower = entry.lower()
|
||||
|
||||
# Check chat_id match
|
||||
if lower.startswith("chat_id:") or lower.startswith("chatid:"):
|
||||
if chat_id is not None:
|
||||
try:
|
||||
allowed_id = int(entry.split(":", 1)[1].strip())
|
||||
if allowed_id == chat_id:
|
||||
return True
|
||||
except ValueError:
|
||||
pass
|
||||
continue
|
||||
|
||||
# Check chat_guid match
|
||||
if lower.startswith("chat_guid:") or lower.startswith("chatguid:"):
|
||||
if chat_guid:
|
||||
allowed_guid = entry.split(":", 1)[1].strip()
|
||||
if allowed_guid == chat_guid:
|
||||
return True
|
||||
continue
|
||||
|
||||
# Normalize and compare handle
|
||||
entry_normalized = normalize_handle(entry)
|
||||
if entry_normalized == sender_normalized:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def _normalize_sender(self, sender: str) -> str:
|
||||
"""Normalize a sender identifier."""
|
||||
return sender if sender.startswith("chat") else normalize_handle(sender)
|
||||
|
||||
def add_allowed_sender(self, sender: str) -> None:
|
||||
"""Add a sender to the allowed list."""
|
||||
normalized = self._normalize_sender(sender)
|
||||
if self.config.allowed_senders is None:
|
||||
self.config.allowed_senders = set()
|
||||
self.config.allowed_senders.add(normalized)
|
||||
logger.info(f"Added allowed sender: {normalized}")
|
||||
|
||||
def remove_allowed_sender(self, sender: str) -> None:
|
||||
"""Remove a sender from the allowed list."""
|
||||
normalized = self._normalize_sender(sender)
|
||||
if self.config.allowed_senders:
|
||||
self.config.allowed_senders.discard(normalized)
|
||||
logger.info(f"Removed allowed sender: {normalized}")
|
||||
|
||||
def clear_allowed_senders(self) -> None:
|
||||
"""Clear allowed list (allow all)."""
|
||||
self.config.allowed_senders = None
|
||||
logger.info("Cleared allowed senders (allowing all)")
|
||||
|
||||
def list_allowed_senders(self) -> list[str]:
|
||||
"""Get current allowed senders."""
|
||||
return list(self.config.allowed_senders) if self.config.allowed_senders else []
|
||||
|
||||
# ── Lifecycle ─────────────────────────────────────────────────
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Initialize and start the channel."""
|
||||
logger.info("Starting iMessage channel (RPC)...")
|
||||
|
||||
self._client = ImsgRpcClient(
|
||||
cli_path=self.config.cli_path,
|
||||
db_path=self.config.db_path,
|
||||
on_notification=self._handle_notification,
|
||||
)
|
||||
|
||||
try:
|
||||
await self._client.start()
|
||||
except Exception as e:
|
||||
raise ChannelError(f"Failed to start imsg: {e}") from e
|
||||
|
||||
# Subscribe to message events
|
||||
try:
|
||||
result = await self._client.request(
|
||||
"watch.subscribe",
|
||||
{"attachments": self.config.include_attachments},
|
||||
)
|
||||
self._subscription_id = result.get("subscription")
|
||||
except Exception as e:
|
||||
await self._client.stop()
|
||||
raise ChannelError(f"Failed to subscribe: {e}") from e
|
||||
|
||||
self._running = True
|
||||
logger.info("iMessage channel started")
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
if self._client and self._subscription_id:
|
||||
try:
|
||||
await self._client.request(
|
||||
"watch.unsubscribe",
|
||||
{"subscription": self._subscription_id},
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
if self._client:
|
||||
await self._client.stop()
|
||||
self._client = None
|
||||
logger.info("iMessage channel stopped")
|
||||
|
||||
# ── Send (template method overrides) ──────────────────────────
|
||||
|
||||
def _resolve_target(self, chat_id: str | None, metadata: dict | None) -> dict:
|
||||
"""Resolve send target from metadata or chat_id string."""
|
||||
meta = metadata or {}
|
||||
for key in ("chat_id", "chat_guid", "chat_identifier"):
|
||||
if meta.get(key):
|
||||
return {key: meta[key]}
|
||||
if chat_id:
|
||||
try:
|
||||
target = parse_target(chat_id)
|
||||
if isinstance(target, ChatIdTarget):
|
||||
return {"chat_id": target.chat_id}
|
||||
elif isinstance(target, ChatGuidTarget):
|
||||
return {"chat_guid": target.chat_guid}
|
||||
elif isinstance(target, ChatIdentifierTarget):
|
||||
return {"chat_identifier": target.chat_identifier}
|
||||
else:
|
||||
return {"to": target.to, "service": target.service.value}
|
||||
except ValueError:
|
||||
return {"to": chat_id}
|
||||
return {}
|
||||
|
||||
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
|
||||
"""Send a single text chunk via iMessage RPC."""
|
||||
if not self._client:
|
||||
raise RuntimeError("iMessage client not running")
|
||||
|
||||
params: dict = {
|
||||
"text": formatted_text,
|
||||
"service": self.config.service,
|
||||
"region": self.config.region,
|
||||
}
|
||||
params.update(self._resolve_target(chat_id, metadata))
|
||||
|
||||
if reply_to:
|
||||
params["reply_to"] = reply_to
|
||||
|
||||
await self._client.request("send", params)
|
||||
|
||||
# ── Retry logic (override base) ───────────────────────────────
|
||||
|
||||
def _format_chunk(self, text: str) -> str:
|
||||
"""iMessage uses plain text; no formatting conversion needed."""
|
||||
return text
|
||||
|
||||
def _extract_retry_after(self, exc: Exception) -> float | None:
|
||||
"""iMessage-specific retry logic.
|
||||
|
||||
RPC errors (e.g. AppleScript failures) are generally not
|
||||
retryable. Transient connection issues get a short retry.
|
||||
"""
|
||||
msg = str(exc).lower()
|
||||
if "not found" in msg or "applescript" in msg or "permission" in msg:
|
||||
return None # not retryable
|
||||
if "timeout" in msg or "connection" in msg:
|
||||
return 1.0
|
||||
return None # default: don't retry RPC errors
|
||||
|
||||
async def _send_media_impl(
|
||||
self,
|
||||
recipient: str,
|
||||
file_path: str,
|
||||
caption: str = "",
|
||||
metadata: dict | None = None,
|
||||
) -> bool:
|
||||
"""Send a media file via iMessage."""
|
||||
if not self._client:
|
||||
return False
|
||||
|
||||
params: dict = {
|
||||
"file": file_path,
|
||||
"service": self.config.service,
|
||||
"region": self.config.region,
|
||||
}
|
||||
|
||||
if caption:
|
||||
params["text"] = caption
|
||||
|
||||
target = self._resolve_target(recipient, metadata)
|
||||
if not target:
|
||||
logger.error("Cannot send media: no recipient")
|
||||
return False
|
||||
params.update(target)
|
||||
|
||||
await self._client.request("send", params)
|
||||
return True
|
||||
@@ -0,0 +1,109 @@
|
||||
"""iMessage environment probe/diagnostics.
|
||||
|
||||
Provides utilities to detect and verify the imsg CLI environment.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import shutil
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProbeResult:
|
||||
"""Result of iMessage environment probe."""
|
||||
|
||||
available: bool = False
|
||||
cli_path: str | None = None
|
||||
cli_version: str | None = None
|
||||
rpc_supported: bool = False
|
||||
error: str | None = None
|
||||
|
||||
|
||||
def find_cli(cli_path: str = "imsg") -> str | None:
|
||||
"""Find the imsg CLI binary.
|
||||
|
||||
Args:
|
||||
cli_path: Path or command name to search
|
||||
|
||||
Returns:
|
||||
Full path to CLI or None if not found
|
||||
"""
|
||||
return shutil.which(cli_path)
|
||||
|
||||
|
||||
async def get_cli_version(cli_path: str) -> str | None:
|
||||
"""Get the imsg CLI version.
|
||||
|
||||
Args:
|
||||
cli_path: Path to CLI binary
|
||||
|
||||
Returns:
|
||||
Version string or None
|
||||
"""
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
cli_path,
|
||||
"--version",
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=5.0)
|
||||
return stdout.decode().strip() or None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
async def check_rpc_support(cli_path: str) -> bool:
|
||||
"""Check if CLI supports RPC mode.
|
||||
|
||||
Args:
|
||||
cli_path: Path to CLI binary
|
||||
|
||||
Returns:
|
||||
True if RPC is supported
|
||||
"""
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
cli_path,
|
||||
"rpc",
|
||||
"--help",
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
await asyncio.wait_for(proc.communicate(), timeout=5.0)
|
||||
return proc.returncode == 0
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
async def probe_imessage(
|
||||
cli_path: str = "imsg",
|
||||
timeout_ms: int = 10000,
|
||||
) -> ProbeResult:
|
||||
"""Probe the iMessage environment.
|
||||
|
||||
Args:
|
||||
cli_path: Path or name of imsg CLI
|
||||
timeout_ms: Timeout in milliseconds
|
||||
|
||||
Returns:
|
||||
ProbeResult with environment details
|
||||
"""
|
||||
result = ProbeResult()
|
||||
|
||||
# Find CLI
|
||||
found_path = find_cli(cli_path)
|
||||
if not found_path:
|
||||
result.error = f"imsg CLI not found: {cli_path}"
|
||||
return result
|
||||
|
||||
result.cli_path = found_path
|
||||
result.available = True
|
||||
|
||||
# Get version
|
||||
result.cli_version = await get_cli_version(found_path)
|
||||
|
||||
# Check RPC support
|
||||
result.rpc_supported = await check_rpc_support(found_path)
|
||||
|
||||
return result
|
||||
@@ -0,0 +1,246 @@
|
||||
"""JSON-RPC client for imsg CLI.
|
||||
|
||||
Communicates with the imsg CLI via JSON-RPC over stdio,
|
||||
similar to OpenClaw's approach.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class RpcError:
|
||||
"""RPC error response."""
|
||||
|
||||
code: int | None = None
|
||||
message: str | None = None
|
||||
data: Any = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class RpcNotification:
|
||||
"""RPC notification (no id, server-initiated)."""
|
||||
|
||||
method: str
|
||||
params: Any = None
|
||||
|
||||
|
||||
class ImsgRpcClient:
|
||||
"""JSON-RPC client for imsg CLI.
|
||||
|
||||
Spawns `imsg rpc` as a subprocess and communicates via stdin/stdout.
|
||||
|
||||
Example:
|
||||
client = ImsgRpcClient(cli_path="/usr/local/bin/imsg")
|
||||
await client.start()
|
||||
result = await client.request("send", {"to": "+1234", "text": "Hello"})
|
||||
await client.stop()
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
cli_path: str = "imsg",
|
||||
db_path: str | None = None,
|
||||
on_notification: Callable[[RpcNotification], None] | None = None,
|
||||
):
|
||||
self.cli_path = cli_path
|
||||
self.db_path = db_path
|
||||
self.on_notification = on_notification
|
||||
|
||||
self._process: asyncio.subprocess.Process | None = None
|
||||
self._next_id = 1
|
||||
self._pending: dict[int, asyncio.Future] = {}
|
||||
self._reader_task: asyncio.Task | None = None
|
||||
self._stderr_task: asyncio.Task | None = None
|
||||
self._closed = asyncio.Event()
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Start the imsg rpc subprocess."""
|
||||
if self._process is not None:
|
||||
return
|
||||
|
||||
args = [self.cli_path, "rpc"]
|
||||
if self.db_path:
|
||||
args.extend(["--db", self.db_path])
|
||||
|
||||
self._process = await asyncio.create_subprocess_exec(
|
||||
*args,
|
||||
stdin=asyncio.subprocess.PIPE,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
|
||||
self._reader_task = asyncio.create_task(self._read_loop())
|
||||
self._stderr_task = asyncio.create_task(self._stderr_loop())
|
||||
logger.info(f"Started imsg rpc (pid={self._process.pid})")
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Stop the imsg rpc subprocess."""
|
||||
if self._process is None:
|
||||
return
|
||||
|
||||
if self._process.stdin:
|
||||
self._process.stdin.close()
|
||||
|
||||
if self._reader_task:
|
||||
self._reader_task.cancel()
|
||||
try:
|
||||
await self._reader_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
if self._stderr_task:
|
||||
self._stderr_task.cancel()
|
||||
try:
|
||||
await self._stderr_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
try:
|
||||
self._process.terminate()
|
||||
await asyncio.wait_for(self._process.wait(), timeout=2.0)
|
||||
except TimeoutError:
|
||||
self._process.kill()
|
||||
await self._process.wait()
|
||||
|
||||
self._fail_all_pending(Exception("RPC client stopped"))
|
||||
self._process = None
|
||||
self._closed.set()
|
||||
logger.info("Stopped imsg rpc")
|
||||
|
||||
async def wait_closed(self) -> None:
|
||||
"""Wait for the client to close."""
|
||||
await self._closed.wait()
|
||||
|
||||
async def request(
|
||||
self,
|
||||
method: str,
|
||||
params: dict[str, Any] | None = None,
|
||||
timeout: float = 10.0,
|
||||
) -> Any:
|
||||
"""Send a JSON-RPC request and wait for response.
|
||||
|
||||
Args:
|
||||
method: RPC method name
|
||||
params: Method parameters
|
||||
timeout: Request timeout in seconds
|
||||
|
||||
Returns:
|
||||
The result from the RPC response
|
||||
|
||||
Raises:
|
||||
Exception: If request fails or times out
|
||||
"""
|
||||
if self._process is None or self._process.stdin is None:
|
||||
raise Exception("RPC client not running")
|
||||
|
||||
request_id = self._next_id
|
||||
self._next_id += 1
|
||||
|
||||
payload = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": request_id,
|
||||
"method": method,
|
||||
"params": params or {},
|
||||
}
|
||||
|
||||
future: asyncio.Future = asyncio.get_event_loop().create_future()
|
||||
self._pending[request_id] = future
|
||||
|
||||
line = json.dumps(payload) + "\n"
|
||||
self._process.stdin.write(line.encode())
|
||||
await self._process.stdin.drain()
|
||||
|
||||
try:
|
||||
return await asyncio.wait_for(future, timeout=timeout)
|
||||
except TimeoutError:
|
||||
self._pending.pop(request_id, None)
|
||||
raise Exception(f"RPC request timeout: {method}") from None
|
||||
|
||||
async def _read_loop(self) -> None:
|
||||
"""Read and process responses from stdout."""
|
||||
if self._process is None or self._process.stdout is None:
|
||||
return
|
||||
|
||||
while True:
|
||||
try:
|
||||
line = await self._process.stdout.readline()
|
||||
if not line:
|
||||
break
|
||||
|
||||
self._handle_line(line.decode().strip())
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception as e:
|
||||
logger.error(f"Error reading from imsg: {e}")
|
||||
break
|
||||
|
||||
self._fail_all_pending(Exception("RPC connection closed"))
|
||||
self._closed.set()
|
||||
|
||||
async def _stderr_loop(self) -> None:
|
||||
"""Log stderr output."""
|
||||
if self._process is None or self._process.stderr is None:
|
||||
return
|
||||
|
||||
while True:
|
||||
try:
|
||||
line = await self._process.stderr.readline()
|
||||
if not line:
|
||||
break
|
||||
logger.warning(f"imsg stderr: {line.decode().strip()}")
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception:
|
||||
break
|
||||
|
||||
def _handle_line(self, line: str) -> None:
|
||||
"""Handle a single JSON-RPC response line."""
|
||||
if not line:
|
||||
return
|
||||
|
||||
try:
|
||||
data = json.loads(line)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Failed to parse RPC response: {e}")
|
||||
return
|
||||
|
||||
# Check if it's a response (has id)
|
||||
if "id" in data and data["id"] is not None:
|
||||
request_id = data["id"]
|
||||
future = self._pending.pop(request_id, None)
|
||||
if future is None:
|
||||
return
|
||||
|
||||
if data.get("error"):
|
||||
error = data["error"]
|
||||
msg = error.get("message", "RPC error")
|
||||
future.set_exception(Exception(msg))
|
||||
else:
|
||||
future.set_result(data.get("result"))
|
||||
return
|
||||
|
||||
# It's a notification
|
||||
if "method" in data:
|
||||
notification = RpcNotification(
|
||||
method=data["method"],
|
||||
params=data.get("params"),
|
||||
)
|
||||
if self.on_notification:
|
||||
try:
|
||||
self.on_notification(notification)
|
||||
except Exception as e:
|
||||
logger.error(f"Notification handler error: {e}")
|
||||
|
||||
def _fail_all_pending(self, error: Exception) -> None:
|
||||
"""Fail all pending requests with an error."""
|
||||
for future in self._pending.values():
|
||||
if not future.done():
|
||||
future.set_exception(error)
|
||||
self._pending.clear()
|
||||
@@ -0,0 +1,89 @@
|
||||
"""iMessage channel server.
|
||||
|
||||
Standalone script to run the iMessage channel with CLI options.
|
||||
|
||||
Usage:
|
||||
python -m EvoScientist.channels.imessage.serve [OPTIONS]
|
||||
|
||||
Examples:
|
||||
# Allow all senders (default)
|
||||
python -m EvoScientist.channels.imessage.serve
|
||||
|
||||
# Only allow specific senders
|
||||
python -m EvoScientist.channels.imessage.serve --allow +1234567890 --allow user@example.com
|
||||
|
||||
# Custom imsg path
|
||||
python -m EvoScientist.channels.imessage.serve --cli-path /usr/local/bin/imsg
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
from ...logging_config import configure_logging_from_settings
|
||||
from ..bus import MessageBus
|
||||
from ..standalone import run_standalone
|
||||
from . import IMessageChannel, IMessageConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="iMessage channel server",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow",
|
||||
action="append",
|
||||
dest="allowed_senders",
|
||||
help="Allowed sender (phone/email). Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cli-path",
|
||||
default="imsg",
|
||||
help="Path to imsg CLI (default: imsg)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--db-path",
|
||||
help="Path to Messages database",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--attachments",
|
||||
action="store_true",
|
||||
help="Include attachments in messages",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--agent",
|
||||
action="store_true",
|
||||
help="Use EvoScientist agent as handler (default: echo)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--thinking",
|
||||
action="store_true",
|
||||
help="Send thinking content as intermediate messages (requires --agent)",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
"""Entry point."""
|
||||
configure_logging_from_settings(default_level=logging.INFO)
|
||||
args = parse_args()
|
||||
|
||||
config = IMessageConfig(
|
||||
cli_path=args.cli_path,
|
||||
db_path=args.db_path,
|
||||
allowed_senders=list(args.allowed_senders) if args.allowed_senders else [],
|
||||
include_attachments=args.attachments,
|
||||
)
|
||||
|
||||
send_thinking = args.thinking and args.agent
|
||||
bus = MessageBus()
|
||||
channel = IMessageChannel(config)
|
||||
|
||||
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,238 @@
|
||||
"""iMessage target parsing and normalization.
|
||||
|
||||
Provides utilities for parsing iMessage targets and normalizing
|
||||
phone numbers and email addresses, similar to OpenClaw's approach.
|
||||
"""
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class IMessageService(Enum):
|
||||
"""iMessage service type."""
|
||||
|
||||
IMESSAGE = "imessage"
|
||||
SMS = "sms"
|
||||
AUTO = "auto"
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatIdTarget:
|
||||
"""Target by chat ID."""
|
||||
|
||||
kind: str = "chat_id"
|
||||
chat_id: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatGuidTarget:
|
||||
"""Target by chat GUID."""
|
||||
|
||||
kind: str = "chat_guid"
|
||||
chat_guid: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatIdentifierTarget:
|
||||
"""Target by chat identifier."""
|
||||
|
||||
kind: str = "chat_identifier"
|
||||
chat_identifier: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class HandleTarget:
|
||||
"""Target by handle (phone/email)."""
|
||||
|
||||
kind: str = "handle"
|
||||
to: str = ""
|
||||
service: IMessageService = IMessageService.AUTO
|
||||
|
||||
|
||||
IMessageTarget = ChatIdTarget | ChatGuidTarget | ChatIdentifierTarget | HandleTarget
|
||||
|
||||
|
||||
# Prefix constants
|
||||
CHAT_ID_PREFIXES = ["chat_id:", "chatid:", "chat:"]
|
||||
CHAT_GUID_PREFIXES = ["chat_guid:", "chatguid:", "guid:"]
|
||||
CHAT_IDENTIFIER_PREFIXES = ["chat_identifier:", "chatidentifier:", "chatident:"]
|
||||
SERVICE_PREFIXES = [
|
||||
("imessage:", IMessageService.IMESSAGE),
|
||||
("sms:", IMessageService.SMS),
|
||||
("auto:", IMessageService.AUTO),
|
||||
]
|
||||
|
||||
|
||||
def normalize_e164(phone: str) -> str | None:
|
||||
"""Normalize phone number to E.164 format.
|
||||
|
||||
Args:
|
||||
phone: Raw phone number string
|
||||
|
||||
Returns:
|
||||
Normalized E.164 format or None if invalid
|
||||
"""
|
||||
# Remove all non-digit characters except leading +
|
||||
cleaned = re.sub(r"[^\d+]", "", phone)
|
||||
|
||||
if not cleaned:
|
||||
return None
|
||||
|
||||
# Already has + prefix
|
||||
if cleaned.startswith("+"):
|
||||
digits = cleaned[1:]
|
||||
if len(digits) >= 10 and len(digits) <= 15:
|
||||
return cleaned
|
||||
return None
|
||||
|
||||
# US/Canada number without country code
|
||||
if len(cleaned) == 10:
|
||||
return f"+1{cleaned}"
|
||||
|
||||
# Has country code
|
||||
if len(cleaned) >= 11 and len(cleaned) <= 15:
|
||||
return f"+{cleaned}"
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def normalize_handle(raw: str) -> str:
|
||||
"""Normalize an iMessage handle (phone or email).
|
||||
|
||||
Args:
|
||||
raw: Raw handle string
|
||||
|
||||
Returns:
|
||||
Normalized handle
|
||||
"""
|
||||
trimmed = raw.strip()
|
||||
if not trimmed:
|
||||
return ""
|
||||
|
||||
lowered = trimmed.lower()
|
||||
|
||||
# Strip service prefixes
|
||||
for prefix, _ in SERVICE_PREFIXES:
|
||||
if lowered.startswith(prefix):
|
||||
return normalize_handle(trimmed[len(prefix) :])
|
||||
|
||||
# Normalize chat_id/chat_guid/chat_identifier prefixes
|
||||
for prefix in CHAT_ID_PREFIXES:
|
||||
if lowered.startswith(prefix):
|
||||
value = trimmed[len(prefix) :].strip()
|
||||
return f"chat_id:{value}"
|
||||
|
||||
for prefix in CHAT_GUID_PREFIXES:
|
||||
if lowered.startswith(prefix):
|
||||
value = trimmed[len(prefix) :].strip()
|
||||
return f"chat_guid:{value}"
|
||||
|
||||
for prefix in CHAT_IDENTIFIER_PREFIXES:
|
||||
if lowered.startswith(prefix):
|
||||
value = trimmed[len(prefix) :].strip()
|
||||
return f"chat_identifier:{value}"
|
||||
|
||||
# Email - lowercase
|
||||
if "@" in trimmed:
|
||||
return trimmed.lower()
|
||||
|
||||
# Phone number - normalize to E.164
|
||||
normalized = normalize_e164(trimmed)
|
||||
if normalized:
|
||||
return normalized
|
||||
|
||||
# Fallback: remove whitespace
|
||||
return re.sub(r"\s+", "", trimmed)
|
||||
|
||||
|
||||
def parse_target(raw: str) -> IMessageTarget:
|
||||
"""Parse an iMessage target string.
|
||||
|
||||
Supports formats:
|
||||
- chat_id:123
|
||||
- chat_guid:abc-def
|
||||
- chat_identifier:iMessage;+;chat123
|
||||
- imessage:+1234567890
|
||||
- sms:+1234567890
|
||||
- +1234567890 (auto service)
|
||||
- email@example.com (auto service)
|
||||
|
||||
Args:
|
||||
raw: Raw target string
|
||||
|
||||
Returns:
|
||||
Parsed IMessageTarget
|
||||
|
||||
Raises:
|
||||
ValueError: If target is invalid
|
||||
"""
|
||||
trimmed = raw.strip()
|
||||
if not trimmed:
|
||||
raise ValueError("iMessage target is required")
|
||||
|
||||
lower = trimmed.lower()
|
||||
|
||||
# Check service prefixes first
|
||||
for prefix, service in SERVICE_PREFIXES:
|
||||
if lower.startswith(prefix):
|
||||
remainder = trimmed[len(prefix) :].strip()
|
||||
if not remainder:
|
||||
raise ValueError(f"{prefix} target is required")
|
||||
|
||||
remainder_lower = remainder.lower()
|
||||
|
||||
# Check if remainder is a chat target
|
||||
is_chat = any(
|
||||
remainder_lower.startswith(p)
|
||||
for p in CHAT_ID_PREFIXES
|
||||
+ CHAT_GUID_PREFIXES
|
||||
+ CHAT_IDENTIFIER_PREFIXES
|
||||
)
|
||||
if is_chat:
|
||||
return parse_target(remainder)
|
||||
|
||||
return HandleTarget(to=remainder, service=service)
|
||||
|
||||
# Check chat_id prefixes
|
||||
for prefix in CHAT_ID_PREFIXES:
|
||||
if lower.startswith(prefix):
|
||||
value = trimmed[len(prefix) :].strip()
|
||||
try:
|
||||
chat_id = int(value)
|
||||
return ChatIdTarget(chat_id=chat_id)
|
||||
except ValueError as e:
|
||||
raise ValueError(f"Invalid chat_id: {value}") from e
|
||||
|
||||
# Check chat_guid prefixes
|
||||
for prefix in CHAT_GUID_PREFIXES:
|
||||
if lower.startswith(prefix):
|
||||
value = trimmed[len(prefix) :].strip()
|
||||
if not value:
|
||||
raise ValueError("chat_guid is required")
|
||||
return ChatGuidTarget(chat_guid=value)
|
||||
|
||||
# Check chat_identifier prefixes
|
||||
for prefix in CHAT_IDENTIFIER_PREFIXES:
|
||||
if lower.startswith(prefix):
|
||||
value = trimmed[len(prefix) :].strip()
|
||||
if not value:
|
||||
raise ValueError("chat_identifier is required")
|
||||
return ChatIdentifierTarget(chat_identifier=value)
|
||||
|
||||
# Default: handle with auto service
|
||||
return HandleTarget(to=trimmed, service=IMessageService.AUTO)
|
||||
|
||||
|
||||
def format_chat_target(chat_id: int | None) -> str:
|
||||
"""Format a chat ID as a target string.
|
||||
|
||||
Args:
|
||||
chat_id: Chat ID number
|
||||
|
||||
Returns:
|
||||
Formatted target string or empty string
|
||||
"""
|
||||
if chat_id is None:
|
||||
return ""
|
||||
return f"chat_id:{chat_id}"
|
||||
@@ -0,0 +1,989 @@
|
||||
"""Composable message processing middleware.
|
||||
|
||||
Each middleware is a standalone class that can be composed into a pipeline.
|
||||
They extract logic that was previously baked into the Channel base class,
|
||||
making it reusable across both legacy and plugin-based channels.
|
||||
|
||||
Also contains the supporting data structures (DedupCache, GroupHistoryBuffer,
|
||||
TypingManager, PairingManager) that were previously in separate files.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import dataclasses
|
||||
import logging
|
||||
import random
|
||||
import time
|
||||
from collections import OrderedDict, deque
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from .base import RawIncoming
|
||||
from .bus.events import InboundMessage, OutboundMessage
|
||||
from .debug import emit_debug_event_if
|
||||
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ── Task cancellation helper ─────────────────────────────────────────
|
||||
|
||||
|
||||
async def _cancel_task(task: asyncio.Task) -> None:
|
||||
"""Cancel an asyncio task and await its completion.
|
||||
|
||||
Suppresses ``CancelledError`` from the cancelled *task* but re-raises
|
||||
if the **current** task was itself cancelled (to avoid swallowing an
|
||||
outer cancellation signal — required for correct behavior on
|
||||
Python 3.12+ where ``_must_cancel`` no longer auto-re-delivers).
|
||||
"""
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
current = asyncio.current_task()
|
||||
if current is not None and current.cancelling() > 0:
|
||||
raise
|
||||
except Exception:
|
||||
pass # Already logged elsewhere; prevent unhandled propagation
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Supporting data structures
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
# ── Dedup cache ──────────────────────────────────────────────────────
|
||||
|
||||
_DEDUP_MAX = 1000
|
||||
_DEDUP_TRIM = 500
|
||||
_DEDUP_TTL = 3600 # 1 hour
|
||||
|
||||
|
||||
class DedupCache:
|
||||
"""Bounded ordered cache with TTL for detecting duplicate message IDs.
|
||||
|
||||
Entries expire after *ttl_seconds* and are pruned lazily on each
|
||||
lookup. When the cache exceeds *max_size* entries it is trimmed
|
||||
down to *trim_to* by evicting the oldest entries. Accessed entries
|
||||
are moved to the end (LRU behavior).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
max_size: int = _DEDUP_MAX,
|
||||
trim_to: int = _DEDUP_TRIM,
|
||||
ttl_seconds: float = _DEDUP_TTL,
|
||||
) -> None:
|
||||
self._seen: OrderedDict[str, float] = OrderedDict()
|
||||
self._max = max_size
|
||||
self._trim = trim_to
|
||||
self._ttl = ttl_seconds
|
||||
|
||||
# ── public API ──────────────────────────────────────────────────
|
||||
|
||||
def is_duplicate(self, msg_id: str) -> bool:
|
||||
"""Return ``True`` if *msg_id* has been seen before.
|
||||
|
||||
First-time IDs are recorded and ``False`` is returned.
|
||||
Empty / falsy IDs are never considered duplicates.
|
||||
Expired entries are pruned before the check.
|
||||
"""
|
||||
if not msg_id:
|
||||
return False
|
||||
|
||||
self._prune()
|
||||
|
||||
if msg_id in self._seen:
|
||||
# LRU: refresh position and timestamp
|
||||
self._seen.move_to_end(msg_id)
|
||||
self._seen[msg_id] = time.monotonic()
|
||||
return True
|
||||
|
||||
self._seen[msg_id] = time.monotonic()
|
||||
if len(self._seen) > self._max:
|
||||
while len(self._seen) > self._trim:
|
||||
self._seen.popitem(last=False)
|
||||
return False
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Remove all entries."""
|
||||
self._seen.clear()
|
||||
|
||||
@property
|
||||
def size(self) -> int:
|
||||
"""Number of entries currently in the cache."""
|
||||
return len(self._seen)
|
||||
|
||||
# ── internal ────────────────────────────────────────────────────
|
||||
|
||||
def _prune(self) -> None:
|
||||
"""Remove entries older than *ttl_seconds*."""
|
||||
cutoff = time.monotonic() - self._ttl
|
||||
# OrderedDict is insertion-ordered; oldest entries are first.
|
||||
while self._seen:
|
||||
_key, ts = next(iter(self._seen.items()))
|
||||
if ts > cutoff:
|
||||
break
|
||||
self._seen.popitem(last=False)
|
||||
|
||||
|
||||
# ── Group history buffer ─────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class HistoryEntry:
|
||||
sender_id: str
|
||||
text: str
|
||||
timestamp: float
|
||||
message_id: str = ""
|
||||
|
||||
|
||||
class GroupHistoryBuffer:
|
||||
"""Per-chat circular buffer of recent messages."""
|
||||
|
||||
def __init__(self, max_per_chat: int = 50, max_age_seconds: int = 3600):
|
||||
self._buffers: dict[str, deque[HistoryEntry]] = {}
|
||||
self._max = max_per_chat
|
||||
self._max_age = max_age_seconds
|
||||
|
||||
def add(self, chat_id: str, entry: HistoryEntry) -> None:
|
||||
"""Add a message to the chat's history buffer."""
|
||||
if chat_id not in self._buffers:
|
||||
self._buffers[chat_id] = deque(maxlen=self._max)
|
||||
self._buffers[chat_id].append(entry)
|
||||
|
||||
def get_recent(self, chat_id: str, limit: int = 20) -> list[HistoryEntry]:
|
||||
"""Get recent messages for context injection, excluding expired ones."""
|
||||
buf = self._buffers.get(chat_id)
|
||||
if not buf:
|
||||
return []
|
||||
now = time.monotonic()
|
||||
recent = [e for e in buf if now - e.timestamp < self._max_age]
|
||||
return recent[-limit:]
|
||||
|
||||
def format_context(self, chat_id: str, limit: int = 20) -> str:
|
||||
"""Format recent messages as context block for the agent."""
|
||||
entries = self.get_recent(chat_id, limit)
|
||||
if not entries:
|
||||
return ""
|
||||
lines = ["[Chat messages since your last reply - for context]"]
|
||||
for e in entries:
|
||||
lines.append(f"[from: {e.sender_id}] {e.text}")
|
||||
lines.append("[/Chat context]")
|
||||
return "\n".join(lines)
|
||||
|
||||
def clear(self, chat_id: str) -> None:
|
||||
"""Clear history for a chat (e.g., after the bot replies)."""
|
||||
self._buffers.pop(chat_id, None)
|
||||
|
||||
|
||||
# ── Typing indicator manager ─────────────────────────────────────────
|
||||
|
||||
|
||||
class TypingManager:
|
||||
"""Manages background typing-indicator loops per chat_id.
|
||||
|
||||
Args:
|
||||
send_action: Async callable that sends a single typing indicator
|
||||
for a given chat_id.
|
||||
interval: Seconds between typing indicator sends.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
send_action: Callable[[str], Awaitable[None]],
|
||||
interval: float = 5.0,
|
||||
debug_trace: bool = False,
|
||||
channel_name: str = "unknown",
|
||||
) -> None:
|
||||
self._send_action = send_action
|
||||
self._interval = interval
|
||||
self._tasks: dict[str, asyncio.Task] = {}
|
||||
self._debug_trace = debug_trace
|
||||
self._channel_name = channel_name
|
||||
|
||||
async def start(self, chat_id: str) -> None:
|
||||
"""Start a background typing-indicator loop for *chat_id*."""
|
||||
await self.stop(chat_id)
|
||||
|
||||
async def _loop() -> None:
|
||||
while True:
|
||||
try:
|
||||
await self._send_action(chat_id)
|
||||
except Exception as exc:
|
||||
_trace_named_event(
|
||||
"typing_error",
|
||||
enabled=self._debug_trace,
|
||||
channel_name=self._channel_name,
|
||||
chat_id=chat_id,
|
||||
error=str(exc),
|
||||
)
|
||||
await asyncio.sleep(self._interval)
|
||||
|
||||
self._tasks[chat_id] = asyncio.create_task(_loop())
|
||||
|
||||
async def stop(self, chat_id: str) -> None:
|
||||
"""Cancel the typing-indicator loop for *chat_id*."""
|
||||
task = self._tasks.pop(chat_id, None)
|
||||
if task:
|
||||
await _cancel_task(task)
|
||||
|
||||
async def stop_all(self) -> None:
|
||||
"""Cancel all active typing-indicator loops."""
|
||||
for cid in list(self._tasks):
|
||||
await self.stop(cid)
|
||||
|
||||
@property
|
||||
def active_chats(self) -> list[str]:
|
||||
"""Return chat_ids with active typing loops."""
|
||||
return list(self._tasks)
|
||||
|
||||
|
||||
# ── Pairing manager ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class PairingRequest:
|
||||
sender_id: str
|
||||
channel: str
|
||||
code: str
|
||||
created_at: float
|
||||
approved: bool = False
|
||||
|
||||
|
||||
class PairingManager:
|
||||
"""Manages DM pairing codes for channel access control."""
|
||||
|
||||
CODE_EXPIRY = 3600 # 1 hour
|
||||
MAX_PENDING = 50 # max pending requests
|
||||
|
||||
def __init__(self):
|
||||
self._pending: dict[str, PairingRequest] = {} # code -> request
|
||||
self._approved: set[str] = set() # "channel:sender_id" keys
|
||||
|
||||
def is_approved(self, channel: str, sender_id: str) -> bool:
|
||||
"""Check if sender is already approved."""
|
||||
return f"{channel}:{sender_id}" in self._approved
|
||||
|
||||
def request_pairing(self, channel: str, sender_id: str) -> str:
|
||||
"""Generate a pairing code for a new sender. Returns the code."""
|
||||
# Check if already has pending request
|
||||
for code, req in list(self._pending.items()):
|
||||
if req.sender_id == sender_id and req.channel == channel:
|
||||
if time.monotonic() - req.created_at < self.CODE_EXPIRY:
|
||||
return code # return existing code
|
||||
else:
|
||||
del self._pending[code]
|
||||
break
|
||||
|
||||
# Cleanup expired
|
||||
self._cleanup_expired()
|
||||
|
||||
# Generate new code
|
||||
code = f"{random.randint(100000, 999999)}"
|
||||
while code in self._pending:
|
||||
code = f"{random.randint(100000, 999999)}"
|
||||
|
||||
self._pending[code] = PairingRequest(
|
||||
sender_id=sender_id,
|
||||
channel=channel,
|
||||
code=code,
|
||||
created_at=time.monotonic(),
|
||||
)
|
||||
_logger.info(f"Pairing code {code} generated for {channel}:{sender_id}")
|
||||
return code
|
||||
|
||||
def approve(self, code: str) -> tuple[bool, str]:
|
||||
"""Approve a pairing code. Returns (success, message)."""
|
||||
req = self._pending.get(code)
|
||||
if not req:
|
||||
return False, f"Unknown code: {code}"
|
||||
if time.monotonic() - req.created_at > self.CODE_EXPIRY:
|
||||
del self._pending[code]
|
||||
return False, f"Code {code} expired"
|
||||
|
||||
key = f"{req.channel}:{req.sender_id}"
|
||||
self._approved.add(key)
|
||||
del self._pending[code]
|
||||
_logger.info(f"Approved pairing for {key}")
|
||||
return True, f"Approved {req.sender_id} on {req.channel}"
|
||||
|
||||
def reject(self, code: str) -> tuple[bool, str]:
|
||||
"""Reject a pairing code."""
|
||||
if code in self._pending:
|
||||
del self._pending[code]
|
||||
return True, f"Rejected code {code}"
|
||||
return False, f"Unknown code: {code}"
|
||||
|
||||
def list_pending(self) -> list[PairingRequest]:
|
||||
"""List all pending (non-expired) requests."""
|
||||
self._cleanup_expired()
|
||||
return list(self._pending.values())
|
||||
|
||||
def _cleanup_expired(self):
|
||||
now = time.monotonic()
|
||||
expired = [
|
||||
c for c, r in self._pending.items() if now - r.created_at > self.CODE_EXPIRY
|
||||
]
|
||||
for c in expired:
|
||||
del self._pending[c]
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Middleware classes
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
# ── Inbound middleware base ──────────────────────────────────────────
|
||||
|
||||
|
||||
class InboundMiddleware:
|
||||
"""Base class for inbound message processing middleware."""
|
||||
|
||||
async def process_inbound(
|
||||
self,
|
||||
raw: RawIncoming,
|
||||
context: dict[str, Any],
|
||||
) -> RawIncoming | None:
|
||||
"""Process an inbound raw message.
|
||||
|
||||
Return the (possibly modified) RawIncoming to continue the
|
||||
pipeline, or ``None`` to drop the message.
|
||||
"""
|
||||
return raw
|
||||
|
||||
|
||||
class OutboundMiddlewareBase:
|
||||
"""Base class for outbound message processing middleware."""
|
||||
|
||||
async def process_outbound(
|
||||
self,
|
||||
message: OutboundMessage,
|
||||
context: dict[str, Any],
|
||||
) -> OutboundMessage | None:
|
||||
"""Process an outbound message.
|
||||
|
||||
Return the (possibly modified) OutboundMessage to continue,
|
||||
or ``None`` to drop it.
|
||||
"""
|
||||
return message
|
||||
|
||||
|
||||
def _debug_trace_enabled(context: dict[str, Any]) -> bool:
|
||||
"""Check whether channel-level debug tracing is enabled for this message."""
|
||||
channel = context.get("channel")
|
||||
if channel is None:
|
||||
return False
|
||||
return channel.is_debug_trace_enabled()
|
||||
|
||||
|
||||
def _ctx_channel_name(context: dict[str, Any]) -> str:
|
||||
"""Extract the channel name from middleware context."""
|
||||
ch = context.get("channel")
|
||||
return getattr(ch, "name", "unknown") if ch else "unknown"
|
||||
|
||||
|
||||
def _trace_context_event(
|
||||
context: dict[str, Any],
|
||||
event: str,
|
||||
**fields: Any,
|
||||
) -> None:
|
||||
"""Emit a middleware trace event using the shared channel context."""
|
||||
emit_debug_event_if(
|
||||
_logger,
|
||||
event,
|
||||
_debug_trace_enabled(context),
|
||||
channel=_ctx_channel_name(context),
|
||||
**fields,
|
||||
)
|
||||
|
||||
|
||||
def _trace_named_event(
|
||||
event: str,
|
||||
*,
|
||||
enabled: bool,
|
||||
channel_name: str,
|
||||
**fields: Any,
|
||||
) -> None:
|
||||
"""Emit a trace event for helpers that already carry trace state."""
|
||||
emit_debug_event_if(
|
||||
_logger,
|
||||
event,
|
||||
enabled,
|
||||
channel=channel_name,
|
||||
**fields,
|
||||
)
|
||||
|
||||
|
||||
# ── Dedup ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class DedupMiddleware(InboundMiddleware):
|
||||
"""Message deduplication using a bounded TTL cache."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
max_size: int = 1000,
|
||||
trim_to: int = 500,
|
||||
ttl_seconds: float = 3600.0,
|
||||
) -> None:
|
||||
self._cache = DedupCache(
|
||||
max_size=max_size,
|
||||
trim_to=trim_to,
|
||||
ttl_seconds=ttl_seconds,
|
||||
)
|
||||
|
||||
async def process_inbound(
|
||||
self,
|
||||
raw: RawIncoming,
|
||||
context: dict[str, Any],
|
||||
) -> RawIncoming | None:
|
||||
if raw.message_id and self._cache.is_duplicate(raw.message_id):
|
||||
_trace_context_event(
|
||||
context,
|
||||
"middleware_dedup_drop",
|
||||
message_id=raw.message_id,
|
||||
sender_id=raw.sender_id,
|
||||
)
|
||||
return None
|
||||
return raw
|
||||
|
||||
|
||||
# ── Debounce ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class DebounceMiddleware:
|
||||
"""Per-sender message batching with configurable timing.
|
||||
|
||||
This middleware collects messages from the same sender and merges
|
||||
them after a debounce delay. It does not follow the simple
|
||||
process_inbound pattern because it needs to buffer across calls.
|
||||
|
||||
Usage: call ``submit()`` for each message; merged results are
|
||||
delivered via the ``on_ready`` callback.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
initial_debounce: float = 2.0,
|
||||
debounce_step: float = 0.5,
|
||||
max_debounce: float = 5.0,
|
||||
on_ready: Callable[[InboundMessage], Any] | None = None,
|
||||
) -> None:
|
||||
self.initial_debounce = initial_debounce
|
||||
self.debounce_step = debounce_step
|
||||
self.max_debounce = max_debounce
|
||||
self.on_ready = on_ready
|
||||
|
||||
self._buffers: dict[str, list[str]] = {}
|
||||
self._metadata: dict[str, dict] = {}
|
||||
self._media: dict[str, list[str]] = {}
|
||||
self._message_ids: dict[str, str] = {}
|
||||
self._tasks: dict[str, asyncio.Task] = {}
|
||||
self._channel_name: str = ""
|
||||
|
||||
def set_channel_name(self, name: str) -> None:
|
||||
self._channel_name = name
|
||||
|
||||
async def submit(self, msg: InboundMessage) -> None:
|
||||
"""Buffer *msg* and schedule flush after debounce delay."""
|
||||
sender = msg.sender_id
|
||||
|
||||
if sender not in self._buffers:
|
||||
self._buffers[sender] = []
|
||||
self._metadata[sender] = msg.metadata
|
||||
self._media[sender] = []
|
||||
self._buffers[sender].append(msg.content)
|
||||
if msg.message_id:
|
||||
self._message_ids[sender] = msg.message_id
|
||||
if msg.media:
|
||||
self._media[sender].extend(msg.media)
|
||||
|
||||
if sender in self._tasks:
|
||||
self._tasks[sender].cancel()
|
||||
|
||||
count = len(self._buffers[sender])
|
||||
wait = min(
|
||||
self.initial_debounce + (count - 1) * self.debounce_step,
|
||||
self.max_debounce,
|
||||
)
|
||||
|
||||
async def _flush(_s: str = sender, _w: float = wait) -> None:
|
||||
await asyncio.sleep(_w)
|
||||
await self._flush_sender(_s)
|
||||
|
||||
self._tasks[sender] = asyncio.create_task(_flush())
|
||||
|
||||
async def _flush_sender(self, sender: str) -> None:
|
||||
messages = self._buffers.pop(sender, [])
|
||||
metadata = self._metadata.pop(sender, None)
|
||||
media = self._media.pop(sender, [])
|
||||
message_id = self._message_ids.pop(sender, "")
|
||||
self._tasks.pop(sender, None)
|
||||
if not messages:
|
||||
return
|
||||
|
||||
merged = "\n".join(messages)
|
||||
chat_id = (metadata or {}).get("chat_id", sender)
|
||||
inbound = InboundMessage(
|
||||
channel=self._channel_name,
|
||||
sender_id=sender,
|
||||
chat_id=str(chat_id),
|
||||
content=merged,
|
||||
media=media,
|
||||
metadata=metadata or {},
|
||||
message_id=message_id,
|
||||
)
|
||||
if self.on_ready:
|
||||
await self.on_ready(inbound)
|
||||
|
||||
async def cancel_all(self) -> None:
|
||||
"""Cancel all pending debounce tasks and await their completion."""
|
||||
tasks = list(self._tasks.values())
|
||||
self._tasks.clear()
|
||||
for task in tasks:
|
||||
await _cancel_task(task)
|
||||
|
||||
|
||||
# ── Chunking ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class ChunkingMiddleware(OutboundMiddlewareBase):
|
||||
"""Auto-split messages respecting format expansion.
|
||||
|
||||
Wraps the existing ``chunking.chunk_text`` utility and the
|
||||
re-splitting logic from ``Channel._prepare_chunks``.
|
||||
"""
|
||||
|
||||
def __init__(self, capabilities: Any) -> None:
|
||||
from .capabilities import ChannelCapabilities
|
||||
|
||||
self._capabilities: ChannelCapabilities = capabilities
|
||||
|
||||
def prepare_chunks(
|
||||
self,
|
||||
content: str,
|
||||
limit: int,
|
||||
format_fn: Callable[[str], str] | None = None,
|
||||
) -> list[tuple[str, str]]:
|
||||
"""Build ``(formatted, raw)`` pairs, re-splitting when needed.
|
||||
|
||||
If *format_fn* is None, formatted == raw.
|
||||
"""
|
||||
from .base import chunk_text
|
||||
|
||||
if format_fn is None:
|
||||
format_fn = lambda t: t # noqa: E731
|
||||
|
||||
raw_chunks = chunk_text(content, limit)
|
||||
pairs: list[tuple[str, str]] = []
|
||||
for raw in raw_chunks:
|
||||
formatted = format_fn(raw)
|
||||
if len(formatted) <= limit:
|
||||
pairs.append((formatted, raw))
|
||||
else:
|
||||
sub_limit = max(limit // 2, 500)
|
||||
for sub_raw in chunk_text(raw, sub_limit):
|
||||
sub_fmt = format_fn(sub_raw)
|
||||
if len(sub_fmt) <= limit:
|
||||
pairs.append((sub_fmt, sub_raw))
|
||||
else:
|
||||
pairs.append((sub_raw, sub_raw))
|
||||
return pairs
|
||||
|
||||
|
||||
# ── Formatting ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class FormattingMiddleware(OutboundMiddlewareBase):
|
||||
"""Markdown -> channel format conversion.
|
||||
|
||||
Uses ``UnifiedFormatter`` configured from capabilities.
|
||||
"""
|
||||
|
||||
def __init__(self, capabilities: Any) -> None:
|
||||
from .capabilities import ChannelCapabilities
|
||||
from .formatter import UnifiedFormatter
|
||||
|
||||
caps: ChannelCapabilities = capabilities
|
||||
self._formatter = UnifiedFormatter.for_channel(caps.format_type)
|
||||
|
||||
def format(self, text: str) -> str:
|
||||
"""Convert text to channel format."""
|
||||
return self._formatter.format(text)
|
||||
|
||||
async def process_outbound(
|
||||
self,
|
||||
message: OutboundMessage,
|
||||
context: dict[str, Any],
|
||||
) -> OutboundMessage | None:
|
||||
formatted = self._formatter.format(message.content)
|
||||
return dataclasses.replace(message, content=formatted)
|
||||
|
||||
|
||||
# ── Retry ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class RetryMiddleware:
|
||||
"""Exponential backoff send retry.
|
||||
|
||||
Wraps ``retry.retry_async`` with channel-appropriate configuration.
|
||||
"""
|
||||
|
||||
def __init__(self, channel_name: str = "unknown") -> None:
|
||||
from .retry import DEFAULT_RETRY, RETRY_PRESETS
|
||||
|
||||
self._config = RETRY_PRESETS.get(channel_name, DEFAULT_RETRY)
|
||||
self._channel_name = channel_name
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
coro_factory: Callable[[], Any],
|
||||
should_retry: Callable[[Exception, int], bool] | None = None,
|
||||
retry_after_s: Callable[[Exception], float | None] | None = None,
|
||||
) -> Any:
|
||||
"""Execute *coro_factory* with retry logic."""
|
||||
from .retry import retry_async
|
||||
|
||||
return await retry_async(
|
||||
coro_factory,
|
||||
config=self._config,
|
||||
should_retry=should_retry or (lambda exc, _: True),
|
||||
retry_after_s=retry_after_s,
|
||||
on_retry=lambda info: _logger.warning(
|
||||
f"{self._channel_name} retry {info.attempt}/{info.max_attempts} "
|
||||
f"in {info.delay_s:.2f}s: {info.error}"
|
||||
),
|
||||
label=f"{self._channel_name}.send",
|
||||
)
|
||||
|
||||
|
||||
# ── Typing ───────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TypingMiddleware:
|
||||
"""Typing indicator management.
|
||||
|
||||
Wraps ``TypingManager`` for use as a standalone middleware component.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
send_typing_fn: Callable[[str], Any],
|
||||
interval: float = 5.0,
|
||||
debug_trace: bool = False,
|
||||
channel_name: str = "unknown",
|
||||
) -> None:
|
||||
self._manager = TypingManager(
|
||||
send_typing_fn,
|
||||
interval=interval,
|
||||
debug_trace=debug_trace,
|
||||
channel_name=channel_name,
|
||||
)
|
||||
|
||||
async def start(self, chat_id: str) -> None:
|
||||
await self._manager.start(chat_id)
|
||||
|
||||
async def stop(self, chat_id: str) -> None:
|
||||
await self._manager.stop(chat_id)
|
||||
|
||||
async def stop_all(self) -> None:
|
||||
await self._manager.stop_all()
|
||||
|
||||
|
||||
# ── ACK Reaction ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class AckReactionMiddleware:
|
||||
"""ACK emoji reaction with configurable scope.
|
||||
|
||||
Scope controls when reactions are sent:
|
||||
- ``"all"``: react to every message
|
||||
- ``"direct"``: react only in DMs
|
||||
- ``"group-all"``: react in group chats (all messages)
|
||||
- ``"group-mentions"``: react in groups only when mentioned
|
||||
- ``"off"``: disable reactions
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
scope: str = "all",
|
||||
emoji: str = "\U0001f440",
|
||||
remove_after_reply: bool = False,
|
||||
send_fn: Callable[[str, str, str], Any] | None = None,
|
||||
remove_fn: Callable[[str, str, str], Any] | None = None,
|
||||
debug_trace: bool = False,
|
||||
channel_name: str = "unknown",
|
||||
) -> None:
|
||||
self.scope = scope
|
||||
self.emoji = emoji
|
||||
self.remove_after_reply = remove_after_reply
|
||||
self._send_fn = send_fn
|
||||
self._remove_fn = remove_fn
|
||||
self._pending: dict[str, str] = {} # chat_id -> message_id
|
||||
self._debug_trace = debug_trace
|
||||
self._channel_name = channel_name
|
||||
|
||||
def should_react(self, *, is_group: bool, was_mentioned: bool) -> bool:
|
||||
if self.scope == "off":
|
||||
return False
|
||||
if self.scope == "all":
|
||||
return True
|
||||
if self.scope == "direct":
|
||||
return not is_group
|
||||
if self.scope == "group-all":
|
||||
return is_group
|
||||
if self.scope == "group-mentions":
|
||||
return is_group and was_mentioned
|
||||
return False
|
||||
|
||||
async def send_ack(self, chat_id: str, message_id: str) -> None:
|
||||
if self._send_fn and message_id:
|
||||
try:
|
||||
await self._send_fn(chat_id, message_id, self.emoji)
|
||||
if self.remove_after_reply:
|
||||
self._pending[chat_id] = message_id
|
||||
except Exception as exc:
|
||||
_trace_named_event(
|
||||
"ack_send_error",
|
||||
enabled=self._debug_trace,
|
||||
channel_name=self._channel_name,
|
||||
chat_id=chat_id,
|
||||
message_id=message_id,
|
||||
error=str(exc),
|
||||
)
|
||||
|
||||
async def remove_ack(self, chat_id: str) -> None:
|
||||
message_id = self._pending.pop(chat_id, None)
|
||||
if message_id and self._remove_fn:
|
||||
try:
|
||||
await self._remove_fn(chat_id, message_id, self.emoji)
|
||||
except Exception as exc:
|
||||
_trace_named_event(
|
||||
"ack_remove_error",
|
||||
enabled=self._debug_trace,
|
||||
channel_name=self._channel_name,
|
||||
chat_id=chat_id,
|
||||
message_id=message_id,
|
||||
error=str(exc),
|
||||
)
|
||||
|
||||
|
||||
# ── Mention Gating ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
class MentionGatingMiddleware(InboundMiddleware):
|
||||
"""Filter messages based on mention policy.
|
||||
|
||||
Policy values:
|
||||
- ``"always"``: require mention in all chats
|
||||
- ``"group"``: require mention only in groups (default)
|
||||
- ``"off"``: never require mention
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
require_mention: str = "group",
|
||||
strip_fn: Callable[[str], str] | None = None,
|
||||
) -> None:
|
||||
self.require_mention = require_mention
|
||||
self._strip_fn = strip_fn
|
||||
|
||||
async def process_inbound(
|
||||
self,
|
||||
raw: RawIncoming,
|
||||
context: dict[str, Any],
|
||||
) -> RawIncoming | None:
|
||||
if not self._should_process(raw):
|
||||
_trace_context_event(
|
||||
context,
|
||||
"middleware_mention_drop",
|
||||
chat_id=raw.chat_id,
|
||||
policy=self.require_mention,
|
||||
)
|
||||
return None
|
||||
# Strip mentions from group messages
|
||||
if raw.is_group and self._strip_fn:
|
||||
raw = dataclasses.replace(raw, text=self._strip_fn(raw.text))
|
||||
return raw
|
||||
|
||||
def _should_process(self, raw: RawIncoming) -> bool:
|
||||
if self.require_mention == "off":
|
||||
return True
|
||||
if self.require_mention == "always":
|
||||
return raw.was_mentioned
|
||||
# "group" — require mention only in groups
|
||||
if not raw.is_group:
|
||||
return True
|
||||
return raw.was_mentioned
|
||||
|
||||
|
||||
# ── AllowList ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class AllowListMiddleware(InboundMiddleware):
|
||||
"""Sender and channel allow-list enforcement."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
allowed_senders: set[str] | None = None,
|
||||
allowed_channels: set[str] | None = None,
|
||||
dm_policy: str = "allowlist",
|
||||
) -> None:
|
||||
self.allowed_senders = allowed_senders
|
||||
self.allowed_channels = allowed_channels
|
||||
self.dm_policy = dm_policy
|
||||
|
||||
async def process_inbound(
|
||||
self,
|
||||
raw: RawIncoming,
|
||||
context: dict[str, Any],
|
||||
) -> RawIncoming | None:
|
||||
# Channel allow-list
|
||||
if self.allowed_channels and str(raw.chat_id) not in self.allowed_channels:
|
||||
_trace_context_event(
|
||||
context,
|
||||
"middleware_allowlist_drop",
|
||||
sender_id=raw.sender_id,
|
||||
chat_id=raw.chat_id,
|
||||
reason="chat_not_allowed",
|
||||
)
|
||||
return None
|
||||
|
||||
# Sender allow-list
|
||||
if not raw.is_group and self.dm_policy == "open":
|
||||
return raw # open DMs bypass sender checks
|
||||
|
||||
if not self._is_sender_allowed(raw.sender_id):
|
||||
_trace_context_event(
|
||||
context,
|
||||
"middleware_allowlist_drop",
|
||||
sender_id=raw.sender_id,
|
||||
chat_id=raw.chat_id,
|
||||
reason="sender_not_allowed",
|
||||
)
|
||||
return None
|
||||
|
||||
return raw
|
||||
|
||||
def _is_sender_allowed(self, sender: str) -> bool:
|
||||
if not self.allowed_senders:
|
||||
return True
|
||||
sender_str = str(sender)
|
||||
if sender_str in self.allowed_senders:
|
||||
return True
|
||||
if "|" in sender_str:
|
||||
for part in sender_str.split("|"):
|
||||
if part and part in self.allowed_senders:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# ── Group History ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class GroupHistoryMiddleware(InboundMiddleware):
|
||||
"""Buffer non-mentioned group messages, inject as context when mentioned."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
max_per_chat: int = 50,
|
||||
max_age_seconds: int = 3600,
|
||||
) -> None:
|
||||
self._buffer = GroupHistoryBuffer(
|
||||
max_per_chat=max_per_chat,
|
||||
max_age_seconds=max_age_seconds,
|
||||
)
|
||||
|
||||
async def process_inbound(
|
||||
self,
|
||||
raw: RawIncoming,
|
||||
context: dict[str, Any],
|
||||
) -> RawIncoming | None:
|
||||
if not raw.is_group:
|
||||
return raw
|
||||
|
||||
# Use monotonic clock for consistent expiry calculation
|
||||
ts = time.monotonic()
|
||||
|
||||
if not raw.was_mentioned:
|
||||
self._buffer.add(
|
||||
raw.chat_id,
|
||||
HistoryEntry(
|
||||
sender_id=raw.sender_id,
|
||||
text=raw.text,
|
||||
timestamp=ts,
|
||||
message_id=raw.message_id,
|
||||
),
|
||||
)
|
||||
# Don't drop here — let MentionGatingMiddleware handle that
|
||||
return raw
|
||||
|
||||
# Mentioned: inject history context
|
||||
history_context = self._buffer.format_context(raw.chat_id)
|
||||
if history_context:
|
||||
raw = dataclasses.replace(
|
||||
raw,
|
||||
text=history_context
|
||||
+ "\n\n[Current message - respond to this]\n"
|
||||
+ raw.text,
|
||||
)
|
||||
self._buffer.clear(raw.chat_id)
|
||||
return raw
|
||||
|
||||
|
||||
# ── Pairing ──────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class PairingMiddleware(InboundMiddleware):
|
||||
"""DM pairing flow management.
|
||||
|
||||
When dm_policy is "pairing", unapproved DM senders receive a
|
||||
pairing code. Approved senders pass through normally.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
channel_name: str,
|
||||
send_response_fn: Callable[[str, str], Any] | None = None,
|
||||
dm_policy: str = "allowlist",
|
||||
) -> None:
|
||||
self._manager = PairingManager()
|
||||
self._channel_name = channel_name
|
||||
self._send_response_fn = send_response_fn
|
||||
self._dm_policy = dm_policy
|
||||
self._background_tasks: set[asyncio.Task] = set()
|
||||
|
||||
async def process_inbound(
|
||||
self,
|
||||
raw: RawIncoming,
|
||||
context: dict[str, Any],
|
||||
) -> RawIncoming | None:
|
||||
if raw.is_group:
|
||||
return raw # pairing only applies to DMs
|
||||
|
||||
if self._dm_policy != "pairing":
|
||||
return raw
|
||||
|
||||
if self._manager.is_approved(self._channel_name, raw.sender_id):
|
||||
return raw
|
||||
|
||||
# Request pairing
|
||||
code = self._manager.request_pairing(self._channel_name, raw.sender_id)
|
||||
if self._send_response_fn:
|
||||
text = f"\U0001f510 Pairing required. Your code: {code}\nThis code expires in 1 hour."
|
||||
task = asyncio.create_task(self._send_response_fn(raw.chat_id, text))
|
||||
# Track the task to prevent GC and handle exceptions
|
||||
self._background_tasks.add(task)
|
||||
task.add_done_callback(self._background_tasks.discard)
|
||||
_trace_context_event(
|
||||
context,
|
||||
"middleware_pairing_required",
|
||||
sender_id=raw.sender_id,
|
||||
)
|
||||
_logger.info(f"Pairing required for {raw.sender_id}, code sent")
|
||||
return None
|
||||
@@ -0,0 +1,342 @@
|
||||
"""Reusable channel mixins for common architecture patterns.
|
||||
|
||||
Three mixins that eliminate boilerplate across channels:
|
||||
|
||||
- ``WebhookMixin`` — aiohttp webhook server + httpx client + token refresh
|
||||
- ``WebSocketMixin`` — WS connect/reconnect/heartbeat loop
|
||||
- ``PollingMixin`` — async poll loop with backoff
|
||||
|
||||
Each mixin works with the Channel base class. Subclasses override
|
||||
a small set of abstract/hook methods to define platform-specific behavior.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Token refresh mixin (shared by Webhook & WebSocket channels)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class TokenMixin:
|
||||
"""Mixin for channels that need OAuth-style token management.
|
||||
|
||||
Subclass must implement ``_fetch_token()`` returning
|
||||
``(access_token, expires_in_seconds)``.
|
||||
"""
|
||||
|
||||
_access_token: str | None = None
|
||||
_token_expires: float = 0
|
||||
_http_client: Any = None # httpx.AsyncClient
|
||||
|
||||
async def _fetch_token(self) -> tuple[str, int]:
|
||||
"""Fetch a new access token. Return (token, expires_in_seconds).
|
||||
|
||||
Must be implemented by the channel.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
async def _refresh_token(self) -> None:
|
||||
token, expire = await self._fetch_token()
|
||||
self._access_token = token
|
||||
self._token_expires = time.monotonic() + expire - 300
|
||||
logger.debug(
|
||||
f"{getattr(self, 'name', '?')} token refreshed, expires in {expire}s"
|
||||
)
|
||||
|
||||
async def _ensure_token(self) -> str:
|
||||
if not self._access_token or time.monotonic() >= self._token_expires:
|
||||
await self._refresh_token()
|
||||
return self._access_token
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Webhook + REST mixin
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class WebhookMixin:
|
||||
"""Mixin for channels that use an HTTP webhook server for inbound
|
||||
and REST API for outbound.
|
||||
|
||||
Provides:
|
||||
- aiohttp web server lifecycle (start/stop)
|
||||
- httpx async client lifecycle
|
||||
- Route registration via ``_webhook_routes()``
|
||||
|
||||
Subclass must implement:
|
||||
- ``_webhook_routes()`` → list of (method, path, handler)
|
||||
- ``_get_webhook_port()`` → int
|
||||
"""
|
||||
|
||||
_http_client: Any = None
|
||||
_runner: Any = None
|
||||
_site: Any = None
|
||||
|
||||
def _get_webhook_port(self) -> int:
|
||||
return getattr(self.config, "webhook_port", 9000)
|
||||
|
||||
def _webhook_routes(self) -> list[tuple[str, str, Any]]:
|
||||
"""Return [(method, path, handler), ...]. Override in subclass."""
|
||||
return []
|
||||
|
||||
async def _start_webhook_server(self) -> None:
|
||||
"""Start aiohttp webhook server + httpx client.
|
||||
|
||||
If ``_shared_webhook_server`` is set (by ChannelManager), the
|
||||
aiohttp server is already running on the shared port — only
|
||||
create the httpx outbound client.
|
||||
"""
|
||||
import httpx
|
||||
|
||||
proxy = getattr(self.config, "proxy", None) or None
|
||||
self._http_client = httpx.AsyncClient(timeout=15, proxy=proxy)
|
||||
|
||||
# Shared webhook mode: routes already registered on shared server
|
||||
if getattr(self, "_shared_webhook_server", None):
|
||||
logger.info(f"{getattr(self, 'name', '?')} using shared webhook server")
|
||||
return
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
app = web.Application()
|
||||
for method, path, handler in self._webhook_routes():
|
||||
if method.upper() == "GET":
|
||||
app.router.add_get(path, handler)
|
||||
else:
|
||||
app.router.add_post(path, handler)
|
||||
|
||||
self._runner = web.AppRunner(app)
|
||||
await self._runner.setup()
|
||||
port = self._get_webhook_port()
|
||||
self._site = web.TCPSite(self._runner, "0.0.0.0", port)
|
||||
await self._site.start()
|
||||
logger.info(f"{getattr(self, 'name', '?')} webhook on port {port}")
|
||||
|
||||
async def _stop_webhook_server(self) -> None:
|
||||
if self._site:
|
||||
await self._site.stop()
|
||||
self._site = None
|
||||
if self._runner:
|
||||
await self._runner.cleanup()
|
||||
self._runner = None
|
||||
if self._http_client:
|
||||
await self._http_client.aclose()
|
||||
self._http_client = None
|
||||
|
||||
async def _api_post(
|
||||
self, url: str, body: dict, headers: dict | None = None
|
||||
) -> dict:
|
||||
"""POST JSON to API, return parsed response. Raises on HTTP error."""
|
||||
resp = await self._http_client.post(url, json=body, headers=headers)
|
||||
data = resp.json()
|
||||
return data
|
||||
|
||||
async def _api_get(self, url: str, headers: dict | None = None) -> dict:
|
||||
resp = await self._http_client.get(url, headers=headers)
|
||||
return resp.json()
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# WebSocket mixin
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class WebSocketMixin:
|
||||
"""Mixin for channels that receive messages via WebSocket.
|
||||
|
||||
Provides:
|
||||
- Connect/reconnect loop with exponential backoff
|
||||
- Heartbeat task management
|
||||
- Message dispatch
|
||||
|
||||
Subclass must implement:
|
||||
- ``_get_ws_url()`` → WebSocket URL to connect to
|
||||
- ``_on_ws_message(data)`` → handle a parsed message dict
|
||||
- ``_on_ws_connected(ws)`` → called after connection (send identify, etc.)
|
||||
|
||||
Optional overrides:
|
||||
- ``_ws_heartbeat_interval`` → seconds between heartbeats (0 = disabled)
|
||||
- ``_on_ws_heartbeat(ws)`` → send heartbeat
|
||||
"""
|
||||
|
||||
_ws_session: Any = None
|
||||
_ws_heartbeat_task: asyncio.Task | None = None
|
||||
_ws_heartbeat_interval: float = 0 # 0 = no heartbeat
|
||||
_ws_reconnect_delay: float = 5.0
|
||||
|
||||
async def _get_ws_url(self) -> str:
|
||||
raise NotImplementedError
|
||||
|
||||
async def _on_ws_connected(self, ws) -> None:
|
||||
"""Called after WebSocket connects. Send identify/auth here."""
|
||||
pass
|
||||
|
||||
async def _on_ws_message(self, data: dict | str) -> None:
|
||||
"""Handle a single WebSocket message."""
|
||||
raise NotImplementedError
|
||||
|
||||
async def _on_ws_heartbeat(self, ws) -> None:
|
||||
"""Send a heartbeat. Override if needed."""
|
||||
pass
|
||||
|
||||
async def _ws_loop(self) -> None:
|
||||
"""Main WebSocket loop with auto-reconnect."""
|
||||
import os
|
||||
|
||||
import aiohttp
|
||||
|
||||
while getattr(self, "_running", False):
|
||||
try:
|
||||
ws_url = await self._get_ws_url()
|
||||
# Resolve proxy: channel config > environment variable
|
||||
proxy = getattr(getattr(self, "config", None), "proxy", None)
|
||||
if not proxy:
|
||||
proxy = (
|
||||
os.environ.get("https_proxy")
|
||||
or os.environ.get("HTTPS_PROXY")
|
||||
or os.environ.get("http_proxy")
|
||||
or os.environ.get("HTTP_PROXY")
|
||||
or None
|
||||
)
|
||||
logger.debug(
|
||||
f"{getattr(self, 'name', '?')} WS connecting to {ws_url[:60]}... proxy={proxy}"
|
||||
)
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.ws_connect(
|
||||
ws_url,
|
||||
proxy=proxy,
|
||||
timeout=aiohttp.ClientWSTimeout(ws_close=30),
|
||||
) as ws:
|
||||
logger.info(f"{getattr(self, 'name', '?')} WebSocket connected")
|
||||
self._ws_session = ws
|
||||
await self._on_ws_connected(ws)
|
||||
|
||||
# Start heartbeat if configured
|
||||
if self._ws_heartbeat_interval > 0:
|
||||
self._ws_heartbeat_task = asyncio.create_task(
|
||||
self._ws_heartbeat_loop(ws)
|
||||
)
|
||||
|
||||
async for msg in ws:
|
||||
if msg.type == aiohttp.WSMsgType.TEXT:
|
||||
try:
|
||||
data = json.loads(msg.data)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
data = msg.data
|
||||
await self._on_ws_message(data)
|
||||
elif msg.type in (
|
||||
aiohttp.WSMsgType.CLOSED,
|
||||
aiohttp.WSMsgType.ERROR,
|
||||
):
|
||||
break
|
||||
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception as e:
|
||||
logger.error(f"{getattr(self, 'name', '?')} WS error: {e}")
|
||||
|
||||
await self._ws_cleanup_heartbeat()
|
||||
self._ws_session = None
|
||||
|
||||
if getattr(self, "_running", False):
|
||||
logger.info(
|
||||
f"{getattr(self, 'name', '?')} reconnecting in {self._ws_reconnect_delay}s..."
|
||||
)
|
||||
await asyncio.sleep(self._ws_reconnect_delay)
|
||||
|
||||
async def _ws_heartbeat_loop(self, ws) -> None:
|
||||
while True:
|
||||
try:
|
||||
await self._on_ws_heartbeat(ws)
|
||||
except Exception:
|
||||
break
|
||||
await asyncio.sleep(self._ws_heartbeat_interval)
|
||||
|
||||
async def _ws_cleanup_heartbeat(self) -> None:
|
||||
if self._ws_heartbeat_task:
|
||||
task = self._ws_heartbeat_task
|
||||
self._ws_heartbeat_task = None
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
current = asyncio.current_task()
|
||||
if current is not None and current.cancelling() > 0:
|
||||
raise
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def _ws_send_json(self, data: dict) -> None:
|
||||
"""Send JSON to the active WebSocket."""
|
||||
if self._ws_session:
|
||||
await self._ws_session.send_str(json.dumps(data))
|
||||
|
||||
async def _stop_ws(self) -> None:
|
||||
await self._ws_cleanup_heartbeat()
|
||||
if self._ws_session:
|
||||
await self._ws_session.close()
|
||||
self._ws_session = None
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# Polling mixin
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class PollingMixin:
|
||||
"""Mixin for channels that poll for new messages.
|
||||
|
||||
Provides:
|
||||
- Poll loop with configurable interval
|
||||
- Error handling + reconnect
|
||||
|
||||
Subclass must implement:
|
||||
- ``_poll_once()`` → fetch and enqueue new messages
|
||||
- ``_get_poll_interval()`` → seconds between polls
|
||||
"""
|
||||
|
||||
_poll_task: asyncio.Task | None = None
|
||||
|
||||
def _get_poll_interval(self) -> float:
|
||||
return getattr(self.config, "poll_interval", 30)
|
||||
|
||||
async def _poll_once(self) -> None:
|
||||
"""Fetch new messages and enqueue them. Override in subclass."""
|
||||
raise NotImplementedError
|
||||
|
||||
async def _start_polling(self) -> None:
|
||||
self._poll_task = asyncio.create_task(self._poll_loop())
|
||||
|
||||
async def _poll_loop(self) -> None:
|
||||
interval = self._get_poll_interval()
|
||||
while getattr(self, "_running", False):
|
||||
try:
|
||||
await self._poll_once()
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception as e:
|
||||
logger.error(f"{getattr(self, 'name', '?')} poll error: {e}")
|
||||
await asyncio.sleep(interval)
|
||||
|
||||
async def _stop_polling(self) -> None:
|
||||
if self._poll_task:
|
||||
task = self._poll_task
|
||||
self._poll_task = None
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
current = asyncio.current_task()
|
||||
if current is not None and current.cancelling() > 0:
|
||||
raise
|
||||
except Exception:
|
||||
pass
|
||||
@@ -0,0 +1,243 @@
|
||||
"""Plugin-based channel interface.
|
||||
|
||||
A ChannelPlugin is a declarative object with optional adapter slots.
|
||||
The framework inspects which slots are filled and auto-assembles
|
||||
the message processing pipeline.
|
||||
|
||||
The ``Channel`` base class extends ``ChannelPlugin``, so all channel
|
||||
implementations are automatically ChannelPlugin instances.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Protocol, runtime_checkable
|
||||
|
||||
from .capabilities import ChannelCapabilities
|
||||
|
||||
# ── Channel metadata ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChannelMeta:
|
||||
"""Channel metadata for registry and UI."""
|
||||
|
||||
id: str
|
||||
label: str
|
||||
description: str = ""
|
||||
docs_path: str = ""
|
||||
system_image: str = "" # icon name
|
||||
|
||||
|
||||
# ── Adapter Protocols (slots) ────────────────────────────────────────
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class ConfigAdapter(Protocol):
|
||||
"""Account configuration management."""
|
||||
|
||||
def list_account_ids(self, config: Any) -> list[str]: ...
|
||||
def resolve_account(self, config: Any, account_id: str | None = None) -> Any: ...
|
||||
def is_enabled(self, account: Any, config: Any) -> bool: ...
|
||||
def is_configured(self, account: Any, config: Any) -> bool: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class SecurityAdapter(Protocol):
|
||||
"""DM policy and security warnings."""
|
||||
|
||||
def resolve_dm_policy(
|
||||
self, ctx: Any
|
||||
) -> str: ... # "open" | "allowlist" | "pairing"
|
||||
def collect_warnings(self, ctx: Any) -> list[str]: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class GroupAdapter(Protocol):
|
||||
"""Per-group policy resolution."""
|
||||
|
||||
def resolve_require_mention(self, ctx: Any) -> bool | None: ...
|
||||
def resolve_tool_policy(self, ctx: Any) -> dict[str, Any] | None: ...
|
||||
def resolve_intro_hint(self, ctx: Any) -> str | None: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class MentionAdapter(Protocol):
|
||||
"""Bot mention detection and stripping."""
|
||||
|
||||
def strip_mentions(self, text: str, ctx: Any) -> str: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class OutboundAdapter(Protocol):
|
||||
"""Outbound message delivery."""
|
||||
|
||||
delivery_mode: str # "direct" | "gateway" | "hybrid"
|
||||
|
||||
async def send_text(self, ctx: Any) -> bool: ...
|
||||
async def send_media(self, ctx: Any) -> bool: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class ThreadingAdapter(Protocol):
|
||||
"""Reply threading behavior."""
|
||||
|
||||
def resolve_reply_to_mode(self, ctx: Any) -> str: ... # "off" | "first" | "all"
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class StreamingAdapter(Protocol):
|
||||
"""Edit-in-place streaming output."""
|
||||
|
||||
async def edit_message(self, chat_id: str, message_id: str, text: str) -> bool: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class DirectoryAdapter(Protocol):
|
||||
"""Contact/group directory queries."""
|
||||
|
||||
async def list_peers(self, ctx: Any) -> list[dict]: ...
|
||||
async def list_groups(self, ctx: Any) -> list[dict]: ...
|
||||
async def list_group_members(self, ctx: Any) -> list[dict]: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class StatusAdapter(Protocol):
|
||||
"""Health probing and status reporting."""
|
||||
|
||||
async def probe_account(self, ctx: Any) -> Any: ...
|
||||
async def audit_account(self, ctx: Any) -> Any: ...
|
||||
def collect_status_issues(self, accounts: list) -> list[dict]: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class HeartbeatAdapter(Protocol):
|
||||
"""Channel heartbeat / readiness checks."""
|
||||
|
||||
async def check_ready(self, ctx: Any) -> tuple[bool, str]: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class ActionsAdapter(Protocol):
|
||||
"""Message actions (react, edit, delete, poll, etc.)."""
|
||||
|
||||
def list_actions(self) -> list[str]: ...
|
||||
async def handle_action(self, action: str, ctx: Any) -> Any: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class PairingAdapter(Protocol):
|
||||
"""DM pairing flow."""
|
||||
|
||||
id_label: str
|
||||
|
||||
def normalize_entry(self, entry: str) -> str: ...
|
||||
async def notify_approval(self, ctx: Any) -> None: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class OnboardingAdapter(Protocol):
|
||||
"""Interactive setup wizard hooks."""
|
||||
|
||||
async def wizard_steps(self, ctx: Any) -> list[dict]: ...
|
||||
async def validate_step(self, step: str, value: Any) -> str | None: ...
|
||||
|
||||
|
||||
# ── Reload policy ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReloadPolicy:
|
||||
"""Declares which config prefixes trigger a channel reload."""
|
||||
|
||||
config_prefixes: list[str] = field(default_factory=list)
|
||||
noop_prefixes: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
# ── ChannelPlugin ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class ChannelPlugin:
|
||||
"""Declarative channel plugin with optional adapter slots.
|
||||
|
||||
Replaces the monolithic Channel base class. Each slot is optional —
|
||||
the framework adapts behavior based on which are present.
|
||||
|
||||
Usage::
|
||||
|
||||
class MyPlugin(ChannelPlugin):
|
||||
id = "my_channel"
|
||||
meta = ChannelMeta(id="my_channel", label="My Channel")
|
||||
capabilities = ChannelCapabilities(...)
|
||||
|
||||
def __init__(self):
|
||||
self.outbound = MyOutboundAdapter()
|
||||
self.config_adapter = MyConfigAdapter()
|
||||
|
||||
async def start(self, config, account_id=None):
|
||||
...
|
||||
|
||||
async def stop(self, account_id=None):
|
||||
...
|
||||
"""
|
||||
|
||||
id: str = ""
|
||||
meta: ChannelMeta | None = None
|
||||
capabilities: ChannelCapabilities = ChannelCapabilities()
|
||||
|
||||
# Optional adapter slots — fill what you need
|
||||
# Default: SingleAccountConfigAdapter so every plugin has multi-account
|
||||
# support out of the box (returns a single "default" account).
|
||||
config_adapter: ConfigAdapter | None = None
|
||||
|
||||
def __init_subclass__(cls, **kwargs: Any) -> None:
|
||||
super().__init_subclass__(**kwargs)
|
||||
|
||||
def __init__(self) -> None:
|
||||
# Provide default SingleAccountConfigAdapter if not overridden
|
||||
if self.config_adapter is None:
|
||||
from .config import SingleAccountConfigAdapter
|
||||
|
||||
self.config_adapter = SingleAccountConfigAdapter()
|
||||
|
||||
security: SecurityAdapter | None = None
|
||||
groups: GroupAdapter | None = None
|
||||
mentions: MentionAdapter | None = None
|
||||
outbound: OutboundAdapter | None = None
|
||||
threading: ThreadingAdapter | None = None
|
||||
streaming: StreamingAdapter | None = None
|
||||
directory: DirectoryAdapter | None = None
|
||||
status: StatusAdapter | None = None
|
||||
heartbeat: HeartbeatAdapter | None = None
|
||||
actions: ActionsAdapter | None = None
|
||||
pairing: PairingAdapter | None = None
|
||||
onboarding: OnboardingAdapter | None = None
|
||||
|
||||
# Lifecycle
|
||||
reload: ReloadPolicy | None = None
|
||||
|
||||
# Connection management
|
||||
async def start(self, config: Any, account_id: str | None = None) -> None:
|
||||
"""Start the channel (or a specific account)."""
|
||||
|
||||
async def stop(self, account_id: str | None = None) -> None:
|
||||
"""Stop the channel (or a specific account)."""
|
||||
|
||||
def filled_slots(self) -> list[str]:
|
||||
"""Return names of adapter slots that are not None."""
|
||||
slot_names = [
|
||||
"config_adapter",
|
||||
"security",
|
||||
"groups",
|
||||
"mentions",
|
||||
"outbound",
|
||||
"threading",
|
||||
"streaming",
|
||||
"directory",
|
||||
"status",
|
||||
"heartbeat",
|
||||
"actions",
|
||||
"pairing",
|
||||
"onboarding",
|
||||
]
|
||||
return [s for s in slot_names if getattr(self, s, None) is not None]
|
||||
@@ -0,0 +1,28 @@
|
||||
"""QQ channel for EvoScientist.
|
||||
|
||||
Uses the official qq-botpy SDK for WebSocket connection.
|
||||
|
||||
Usage in config:
|
||||
channel_enabled = "qq"
|
||||
qq_app_id = "your_app_id"
|
||||
qq_app_secret = "your_app_secret"
|
||||
"""
|
||||
|
||||
from ..channel_manager import _parse_csv, register_channel
|
||||
from .channel import QQChannel, QQConfig
|
||||
|
||||
__all__ = ["QQChannel", "QQConfig"]
|
||||
|
||||
|
||||
def create_from_config(config) -> QQChannel:
|
||||
allowed = _parse_csv(config.qq_allowed_senders)
|
||||
return QQChannel(
|
||||
QQConfig(
|
||||
app_id=config.qq_app_id,
|
||||
app_secret=config.qq_app_secret,
|
||||
allowed_senders=allowed,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
register_channel("qq", create_from_config)
|
||||
@@ -0,0 +1,374 @@
|
||||
"""QQ channel implementation using botpy SDK."""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections import deque
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import ClassVar
|
||||
|
||||
from ..base import Channel, ChannelError, RawIncoming
|
||||
from ..capabilities import QQ as QQ_CAPS
|
||||
from ..config import BaseChannelConfig
|
||||
from ..formatter import UnifiedFormatter
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
try:
|
||||
import botpy
|
||||
from botpy.message import C2CMessage, GroupMessage
|
||||
|
||||
QQ_AVAILABLE = True
|
||||
except ImportError:
|
||||
QQ_AVAILABLE = False
|
||||
botpy = None
|
||||
C2CMessage = None
|
||||
GroupMessage = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class QQConfig(BaseChannelConfig):
|
||||
app_id: str = ""
|
||||
app_secret: str = ""
|
||||
text_chunk_limit: int = 4096
|
||||
|
||||
|
||||
def _make_bot_class(channel: "QQChannel") -> "type[botpy.Client]":
|
||||
"""Create a botpy Client subclass bound to the given channel."""
|
||||
intents = botpy.Intents(public_messages=True, direct_message=True)
|
||||
|
||||
class _Bot(botpy.Client):
|
||||
def __init__(self):
|
||||
super().__init__(intents=intents)
|
||||
|
||||
async def on_ready(self):
|
||||
logger.info(f"QQ bot ready: {self.robot.name}")
|
||||
|
||||
async def on_c2c_message_create(self, message: "C2CMessage"):
|
||||
await channel._on_msg(message, "c2c")
|
||||
|
||||
async def on_group_at_message_create(self, message: "GroupMessage"):
|
||||
await channel._on_msg(message, "group")
|
||||
|
||||
return _Bot
|
||||
|
||||
|
||||
class QQChannel(Channel):
|
||||
"""QQ channel using botpy SDK."""
|
||||
|
||||
name = "qq"
|
||||
|
||||
capabilities = QQ_CAPS
|
||||
_ready_attrs = ("_client", "_running")
|
||||
_non_retryable_patterns = ()
|
||||
_mention_pattern = r"@\S+\s*"
|
||||
_mention_strip_count = 1
|
||||
_markdown_fallback_exc_types: ClassVar[tuple[type[Exception], ...]] = (
|
||||
TypeError,
|
||||
ValueError,
|
||||
)
|
||||
|
||||
def __init__(self, config: QQConfig):
|
||||
super().__init__(config)
|
||||
self._client: botpy.Client | None = None
|
||||
self._bot_task: asyncio.Task | None = None
|
||||
self._processed_ids: deque = deque(maxlen=1000)
|
||||
self._msg_seq: dict[str, int] = {} # msg_id -> next seq number
|
||||
self._msg_seq_order: deque = deque(maxlen=500)
|
||||
self._msg_seq_ids: set[str] = set() # companion set for O(1) lookup
|
||||
self._plain_formatter = UnifiedFormatter.for_channel("plain")
|
||||
|
||||
# ── Lifecycle ─────────────────────────────────────────────────
|
||||
|
||||
async def start(self) -> None:
|
||||
if not QQ_AVAILABLE:
|
||||
raise ChannelError("QQ SDK not installed. Run: pip install qq-botpy")
|
||||
if not self.config.app_id or not self.config.app_secret:
|
||||
raise ChannelError("QQ app_id and app_secret are required")
|
||||
self._running = True
|
||||
BotClass = _make_bot_class(self)
|
||||
self._client = BotClass()
|
||||
self._bot_task = asyncio.create_task(self._run_bot())
|
||||
logger.info("QQ channel starting...")
|
||||
|
||||
async def _run_bot(self) -> None:
|
||||
try:
|
||||
await self._client.start(
|
||||
appid=self.config.app_id, secret=self.config.app_secret
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"QQ auth failed: {e}")
|
||||
self._running = False
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
self._running = False
|
||||
if self._bot_task:
|
||||
self._bot_task.cancel()
|
||||
try:
|
||||
await self._bot_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self._client = None
|
||||
logger.info("QQ channel stopped")
|
||||
|
||||
# ── Incoming ──────────────────────────────────────────────────
|
||||
|
||||
async def _on_msg(self, message, msg_type: str) -> None:
|
||||
try:
|
||||
if message.id in self._processed_ids:
|
||||
return
|
||||
self._processed_ids.append(message.id)
|
||||
|
||||
author = message.author
|
||||
content = (message.content or "").strip()
|
||||
|
||||
if msg_type == "c2c":
|
||||
sender_id = str(getattr(author, "user_openid", ""))
|
||||
chat_id = sender_id
|
||||
else:
|
||||
sender_id = str(getattr(author, "member_openid", ""))
|
||||
chat_id = str(getattr(message, "group_openid", ""))
|
||||
|
||||
# Handle attachments (images, files, audio, video)
|
||||
annotations: list[str] = []
|
||||
media_paths: list[str] = []
|
||||
attachments = getattr(message, "attachments", None) or []
|
||||
for att in attachments:
|
||||
url = getattr(att, "url", "") or ""
|
||||
filename = getattr(att, "filename", "attachment") or "attachment"
|
||||
content_type = getattr(att, "content_type", "") or ""
|
||||
if url:
|
||||
local, ann = await self._download_attachment(
|
||||
url,
|
||||
f"qq_{filename}",
|
||||
)
|
||||
if local:
|
||||
media_paths.append(local)
|
||||
if ann:
|
||||
annotations.append(ann)
|
||||
else:
|
||||
annotations.append(f"[{content_type or 'attachment'}: {filename}]")
|
||||
|
||||
if not content and not media_paths and not annotations:
|
||||
return
|
||||
|
||||
await self._enqueue_raw(
|
||||
RawIncoming(
|
||||
sender_id=sender_id,
|
||||
chat_id=chat_id,
|
||||
text=content,
|
||||
media_files=media_paths,
|
||||
content_annotations=annotations,
|
||||
timestamp=datetime.now(),
|
||||
message_id=message.id,
|
||||
is_group=(msg_type == "group"),
|
||||
was_mentioned=True,
|
||||
metadata={
|
||||
"chat_id": chat_id,
|
||||
"msg_type": msg_type,
|
||||
"event_id": message.id,
|
||||
"backend": "qq",
|
||||
},
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error handling QQ message: {e}")
|
||||
|
||||
# ── Send ──────────────────────────────────────────────────────
|
||||
|
||||
def _next_msg_seq(self, msg_id: str) -> int:
|
||||
"""Return the next msg_seq for *msg_id* and increment the counter."""
|
||||
seq = self._msg_seq.get(msg_id, 1)
|
||||
self._msg_seq[msg_id] = seq + 1
|
||||
if msg_id not in self._msg_seq_ids:
|
||||
self._msg_seq_order.append(msg_id)
|
||||
self._msg_seq_ids.add(msg_id)
|
||||
if len(self._msg_seq_order) > 500:
|
||||
oldest = self._msg_seq_order.popleft()
|
||||
self._msg_seq_ids.discard(oldest)
|
||||
self._msg_seq.pop(oldest, None)
|
||||
return seq
|
||||
|
||||
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
|
||||
if not self._client:
|
||||
raise ChannelError("QQ client not initialized")
|
||||
msg_type = (metadata or {}).get("msg_type", "c2c")
|
||||
msg_id = (metadata or {}).get("event_id", "")
|
||||
seq = self._next_msg_seq(msg_id)
|
||||
try:
|
||||
await self._post_markdown_message(chat_id, raw_text, msg_type, msg_id, seq)
|
||||
return
|
||||
except Exception as exc:
|
||||
if not self._should_fallback_to_plain_text(exc):
|
||||
raise
|
||||
self._record_markdown_fallback(chat_id, raw_text, exc)
|
||||
logger.debug("QQ markdown send failed, falling back to plain text: %s", exc)
|
||||
|
||||
plain_text = self._plain_formatter.format(raw_text)
|
||||
await self._post_plain_message(chat_id, plain_text, msg_type, msg_id, seq)
|
||||
|
||||
def _should_fallback_to_plain_text(self, exc: Exception) -> bool:
|
||||
"""Return True only for markdown compatibility/validation failures."""
|
||||
if isinstance(exc, self._markdown_fallback_exc_types):
|
||||
return True
|
||||
|
||||
msg = str(exc).lower()
|
||||
compatibility_tokens = ("unsupported", "unexpected", "unknown", "invalid")
|
||||
return (
|
||||
"unexpected keyword argument" in msg
|
||||
or (
|
||||
"markdown" in msg
|
||||
and any(token in msg for token in compatibility_tokens)
|
||||
)
|
||||
or (
|
||||
"msg_type" in msg
|
||||
and any(token in msg for token in compatibility_tokens)
|
||||
)
|
||||
)
|
||||
|
||||
def _record_markdown_fallback(
|
||||
self,
|
||||
chat_id: str,
|
||||
raw_text: str,
|
||||
exc: Exception,
|
||||
) -> None:
|
||||
"""Emit optional debug trace for markdown fallback without blocking send."""
|
||||
trace_event = getattr(self, "_trace_event", None)
|
||||
if not callable(trace_event):
|
||||
return
|
||||
try:
|
||||
trace_event(
|
||||
"outbound_format_fallback",
|
||||
chat_id=chat_id,
|
||||
error=str(exc),
|
||||
formatted_len=len(raw_text),
|
||||
raw_len=len(raw_text),
|
||||
)
|
||||
except Exception as trace_exc:
|
||||
logger.debug("QQ fallback trace failed: %s", trace_exc)
|
||||
|
||||
async def _post_markdown_message(
|
||||
self,
|
||||
chat_id: str,
|
||||
text: str,
|
||||
msg_type: str,
|
||||
msg_id: str,
|
||||
seq: int,
|
||||
) -> None:
|
||||
payload = {
|
||||
"msg_type": 2,
|
||||
"markdown": {"content": text},
|
||||
"msg_id": msg_id,
|
||||
"msg_seq": seq,
|
||||
}
|
||||
if msg_type == "group":
|
||||
await self._client.api.post_group_message(
|
||||
group_openid=chat_id,
|
||||
**payload,
|
||||
)
|
||||
else:
|
||||
await self._client.api.post_c2c_message(
|
||||
openid=chat_id,
|
||||
**payload,
|
||||
)
|
||||
|
||||
async def _post_plain_message(
|
||||
self,
|
||||
chat_id: str,
|
||||
text: str,
|
||||
msg_type: str,
|
||||
msg_id: str,
|
||||
seq: int,
|
||||
) -> None:
|
||||
payload = {
|
||||
"msg_type": 0,
|
||||
"content": text,
|
||||
"msg_id": msg_id,
|
||||
"msg_seq": seq,
|
||||
}
|
||||
if msg_type == "group":
|
||||
await self._client.api.post_group_message(
|
||||
group_openid=chat_id,
|
||||
**payload,
|
||||
)
|
||||
else:
|
||||
await self._client.api.post_c2c_message(
|
||||
openid=chat_id,
|
||||
**payload,
|
||||
)
|
||||
|
||||
# _send_typing_action: inherited no-op (QQ Bot API has no typing indicator)
|
||||
|
||||
# ── Media send ────────────────────────────────────────────────
|
||||
|
||||
# qq-botpy file_type constants: 1=image, 2=video, 3=audio
|
||||
_FILE_TYPE_MAP: ClassVar[dict[str, int]] = {
|
||||
".jpg": 1,
|
||||
".jpeg": 1,
|
||||
".png": 1,
|
||||
".gif": 1,
|
||||
".webp": 1,
|
||||
".bmp": 1,
|
||||
".mp4": 2,
|
||||
".mov": 2,
|
||||
".avi": 2,
|
||||
".mp3": 3,
|
||||
".ogg": 3,
|
||||
".m4a": 3,
|
||||
".wav": 3,
|
||||
".silk": 3,
|
||||
}
|
||||
|
||||
async def _send_media_impl(
|
||||
self,
|
||||
recipient: str,
|
||||
file_path: str,
|
||||
caption: str = "",
|
||||
metadata: dict | None = None,
|
||||
) -> bool:
|
||||
"""Send a media file through QQ Bot API.
|
||||
|
||||
Uses post_group_file / post_c2c_file with a URL. Local files
|
||||
without a public URL are not supported — falls back to a text hint.
|
||||
"""
|
||||
if not self._client:
|
||||
raise ChannelError("QQ client not initialized")
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
chat_id = self._resolve_media_chat_id(recipient, metadata)
|
||||
msg_type = (metadata or {}).get("msg_type", "c2c")
|
||||
ext = Path(file_path).suffix.lower()
|
||||
file_type = self._FILE_TYPE_MAP.get(ext, 1) # default to image
|
||||
|
||||
# qq-botpy file API requires a URL, not a local path
|
||||
is_url = file_path.startswith("http://") or file_path.startswith("https://")
|
||||
if not is_url:
|
||||
# Fallback: send text hint for local files
|
||||
name = Path(file_path).name
|
||||
hint = f"[文件] {name}" + (f"\n{caption}" if caption else "")
|
||||
await self._send_chunk(chat_id, hint, hint, None, metadata or {})
|
||||
return True
|
||||
|
||||
try:
|
||||
if msg_type == "group":
|
||||
await self._client.api.post_group_file(
|
||||
group_openid=chat_id,
|
||||
file_type=file_type,
|
||||
url=file_path,
|
||||
srv_send_msg=True,
|
||||
)
|
||||
else:
|
||||
await self._client.api.post_c2c_file(
|
||||
openid=chat_id,
|
||||
file_type=file_type,
|
||||
url=file_path,
|
||||
srv_send_msg=True,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"QQ media send failed: {e}")
|
||||
return False
|
||||
|
||||
if caption:
|
||||
await self._send_chunk(chat_id, caption, caption, None, metadata or {})
|
||||
return True
|
||||
@@ -0,0 +1,37 @@
|
||||
"""QQ Bot credential validation."""
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
QQ_TOKEN_URL = "https://bots.qq.com/app/getAppAccessToken"
|
||||
|
||||
|
||||
async def validate_qq(
|
||||
app_id: str,
|
||||
app_secret: str,
|
||||
) -> tuple[bool, str]:
|
||||
"""Validate QQ Bot credentials by fetching an access token.
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, message).
|
||||
"""
|
||||
if not app_id or not app_secret:
|
||||
return False, "app_id and app_secret are required"
|
||||
|
||||
try:
|
||||
import httpx
|
||||
except ImportError:
|
||||
return False, "httpx not installed"
|
||||
|
||||
body = {"appId": app_id, "clientSecret": app_secret}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(QQ_TOKEN_URL, json=body, timeout=10)
|
||||
data = resp.json()
|
||||
if data.get("access_token"):
|
||||
return True, "QQ Bot credentials valid"
|
||||
return False, f"Error: {data.get('message', data)}"
|
||||
except Exception as e:
|
||||
return False, f"Error: {e}"
|
||||
@@ -0,0 +1,84 @@
|
||||
"""QQ channel server.
|
||||
|
||||
Standalone script to run the QQ channel with CLI options.
|
||||
|
||||
Usage:
|
||||
python -m EvoScientist.channels.qq.serve --app-id ID --app-secret SECRET [OPTIONS]
|
||||
|
||||
Examples:
|
||||
# Basic usage
|
||||
python -m EvoScientist.channels.qq.serve --app-id ID --app-secret SECRET
|
||||
|
||||
# Sandbox mode with allowed senders
|
||||
python -m EvoScientist.channels.qq.serve --app-id ID --app-secret SECRET --allow user123
|
||||
|
||||
# With agent and thinking
|
||||
python -m EvoScientist.channels.qq.serve --app-id ID --app-secret SECRET --agent --thinking
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
from ...logging_config import configure_logging_from_settings
|
||||
from ..bus import MessageBus
|
||||
from ..standalone import run_standalone
|
||||
from .channel import QQChannel, QQConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="QQ channel server",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--app-id",
|
||||
required=True,
|
||||
help="QQ bot app ID",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--app-secret",
|
||||
required=True,
|
||||
help="QQ bot app secret",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow",
|
||||
action="append",
|
||||
dest="allowed_senders",
|
||||
help="Allowed sender (QQ user ID). Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--agent",
|
||||
action="store_true",
|
||||
help="Use EvoScientist agent as handler (default: echo)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--thinking",
|
||||
action="store_true",
|
||||
help="Send thinking content as intermediate messages (requires --agent)",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
"""Entry point."""
|
||||
configure_logging_from_settings(default_level=logging.INFO)
|
||||
args = parse_args()
|
||||
|
||||
config = QQConfig(
|
||||
app_id=args.app_id,
|
||||
app_secret=args.app_secret,
|
||||
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
|
||||
)
|
||||
|
||||
send_thinking = args.thinking and args.agent
|
||||
bus = MessageBus()
|
||||
channel = QQChannel(config)
|
||||
|
||||
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,114 @@
|
||||
"""Configurable exponential-backoff retry for async callables."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import random
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
from typing import TypeVar
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
@dataclass
|
||||
class RetryConfig:
|
||||
"""Configuration for retry behavior."""
|
||||
|
||||
attempts: int = 3
|
||||
min_delay_s: float = 0.3
|
||||
max_delay_s: float = 30.0
|
||||
jitter: float = 0.1 # ±10 % random offset
|
||||
|
||||
|
||||
@dataclass
|
||||
class RetryInfo:
|
||||
"""Information passed to the *on_retry* callback."""
|
||||
|
||||
attempt: int
|
||||
max_attempts: int
|
||||
delay_s: float
|
||||
error: Exception
|
||||
label: str | None = None
|
||||
|
||||
|
||||
async def retry_async(
|
||||
fn: Callable[[], Awaitable[T]],
|
||||
config: RetryConfig | None = None,
|
||||
*,
|
||||
should_retry: Callable[[Exception, int], bool] | None = None,
|
||||
retry_after_s: Callable[[Exception], float | None] | None = None,
|
||||
on_retry: Callable[[RetryInfo], None] | None = None,
|
||||
label: str | None = None,
|
||||
) -> T:
|
||||
"""Execute *fn* with exponential-backoff retry.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
fn:
|
||||
Zero-argument async factory — called on every attempt so the
|
||||
awaitable is always fresh.
|
||||
"""
|
||||
if config is None:
|
||||
config = RetryConfig()
|
||||
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(1, config.attempts + 1):
|
||||
try:
|
||||
return await fn()
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
|
||||
if attempt >= config.attempts:
|
||||
raise
|
||||
|
||||
if should_retry is not None and not should_retry(exc, attempt):
|
||||
raise
|
||||
|
||||
# Compute delay
|
||||
server_delay: float | None = None
|
||||
if retry_after_s is not None:
|
||||
server_delay = retry_after_s(exc)
|
||||
|
||||
if server_delay is not None:
|
||||
base_delay = max(server_delay, config.min_delay_s)
|
||||
else:
|
||||
base_delay = config.min_delay_s * (2 ** (attempt - 1))
|
||||
|
||||
# Apply jitter
|
||||
jittered = base_delay * (1 + random.uniform(-config.jitter, config.jitter))
|
||||
|
||||
# Clamp to [min_delay_s, max_delay_s]
|
||||
delay = max(config.min_delay_s, min(jittered, config.max_delay_s))
|
||||
|
||||
if on_retry is not None:
|
||||
on_retry(
|
||||
RetryInfo(
|
||||
attempt=attempt,
|
||||
max_attempts=config.attempts,
|
||||
delay_s=delay,
|
||||
error=exc,
|
||||
label=label,
|
||||
)
|
||||
)
|
||||
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
# Should never reach here, but satisfy the type checker.
|
||||
assert last_exc is not None
|
||||
raise last_exc
|
||||
|
||||
|
||||
# ── Presets ──────────────────────────────────────────────────────────
|
||||
|
||||
TELEGRAM_RETRY = RetryConfig(attempts=3, min_delay_s=0.4, max_delay_s=30.0, jitter=0.1)
|
||||
DEFAULT_RETRY = RetryConfig()
|
||||
|
||||
# Discord, Slack, Teams, Feishu all use the same config (attempts=3,
|
||||
# min_delay_s=0.5, max_delay_s=30.0, jitter=0.1) — close enough to
|
||||
# DEFAULT_RETRY that separate presets add no value. Channels that
|
||||
# don't appear in RETRY_PRESETS already fall back to DEFAULT_RETRY.
|
||||
|
||||
RETRY_PRESETS: dict[str, RetryConfig] = {
|
||||
"telegram": TELEGRAM_RETRY,
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
"""Signal channel for EvoScientist.
|
||||
|
||||
Uses signal-cli in JSON RPC mode — no public IP needed.
|
||||
|
||||
Usage in config:
|
||||
channel_enabled = "signal"
|
||||
signal_phone_number = "+1234567890"
|
||||
"""
|
||||
|
||||
from ..channel_manager import _parse_csv, register_channel
|
||||
from .channel import SignalChannel, SignalConfig
|
||||
|
||||
__all__ = ["SignalChannel", "SignalConfig"]
|
||||
|
||||
|
||||
def create_from_config(config) -> SignalChannel:
|
||||
allowed = _parse_csv(config.signal_allowed_senders)
|
||||
return SignalChannel(
|
||||
SignalConfig(
|
||||
phone_number=config.signal_phone_number,
|
||||
cli_path=config.signal_cli_path,
|
||||
config_dir=config.signal_config_dir or None,
|
||||
rpc_port=config.signal_rpc_port,
|
||||
allowed_senders=allowed,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
register_channel("signal", create_from_config)
|
||||
@@ -0,0 +1,510 @@
|
||||
"""Signal channel implementation using signal-cli JSON RPC."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import subprocess
|
||||
from collections import deque
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from ..base import Channel, ChannelError, RawIncoming
|
||||
from ..capabilities import SIGNAL as SIGNAL_CAPS
|
||||
from ..config import BaseChannelConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class SignalConfig(BaseChannelConfig):
|
||||
phone_number: str = ""
|
||||
cli_path: str = "signal-cli"
|
||||
config_dir: str | None = None
|
||||
rpc_port: int = 7583
|
||||
text_chunk_limit: int = 4096
|
||||
|
||||
|
||||
class SignalChannel(Channel):
|
||||
"""Signal channel using signal-cli JSON RPC."""
|
||||
|
||||
name = "signal"
|
||||
|
||||
capabilities = SIGNAL_CAPS
|
||||
_non_retryable_patterns = ("unregistered", "auth")
|
||||
|
||||
def __init__(self, config: SignalConfig):
|
||||
super().__init__(config)
|
||||
self._reader: asyncio.StreamReader | None = None
|
||||
self._writer: asyncio.StreamWriter | None = None
|
||||
self._rpc_id = 0
|
||||
self._daemon_proc = None
|
||||
# Pending RPC responses: rpc_id -> Future
|
||||
self._pending_rpcs: dict[int, asyncio.Future] = {}
|
||||
# Cache message_id → sender for reaction targetAuthor (bounded)
|
||||
self._msg_senders: dict[str, str] = {}
|
||||
self._msg_senders_order: deque = deque(maxlen=200)
|
||||
self._listen_task: asyncio.Task | None = None
|
||||
|
||||
async def start(self) -> None:
|
||||
if not self.config.phone_number:
|
||||
raise ChannelError("Signal phone_number is required")
|
||||
|
||||
# Try to start signal-cli daemon if not already running
|
||||
await self._ensure_daemon()
|
||||
|
||||
try:
|
||||
# Connect to JSON RPC socket
|
||||
await self._connect()
|
||||
except Exception:
|
||||
# If connect fails after daemon was started, clean up the daemon
|
||||
await self._cleanup()
|
||||
raise
|
||||
|
||||
self._running = True
|
||||
logger.info(f"Signal channel started (phone: {self.config.phone_number})")
|
||||
|
||||
# Listen for incoming messages in background task
|
||||
# (start() must return so that run() can iterate receive())
|
||||
self._listen_task = asyncio.create_task(self._listen_loop())
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
if self._listen_task:
|
||||
self._listen_task.cancel()
|
||||
self._listen_task = None
|
||||
# Cancel any pending RPC futures
|
||||
for fut in self._pending_rpcs.values():
|
||||
if not fut.done():
|
||||
fut.cancel()
|
||||
self._pending_rpcs.clear()
|
||||
if self._writer:
|
||||
self._writer.close()
|
||||
try:
|
||||
await self._writer.wait_closed()
|
||||
except Exception:
|
||||
pass
|
||||
self._writer = None
|
||||
self._reader = None
|
||||
if self._daemon_proc:
|
||||
self._daemon_proc.terminate()
|
||||
self._daemon_proc = None
|
||||
logger.info("Signal channel stopped")
|
||||
|
||||
async def _ensure_daemon(self) -> None:
|
||||
"""Start signal-cli daemon if not already running."""
|
||||
try:
|
||||
_reader, writer = await asyncio.wait_for(
|
||||
asyncio.open_connection("localhost", self.config.rpc_port),
|
||||
timeout=2,
|
||||
)
|
||||
writer.close()
|
||||
await writer.wait_closed()
|
||||
logger.info("signal-cli daemon already running")
|
||||
return
|
||||
except (TimeoutError, ConnectionRefusedError, OSError):
|
||||
pass
|
||||
|
||||
# Start daemon
|
||||
cmd = [self.config.cli_path, "-u", self.config.phone_number]
|
||||
if self.config.config_dir:
|
||||
cmd.extend(["--config", self.config.config_dir])
|
||||
cmd.extend(
|
||||
[
|
||||
"daemon",
|
||||
"--tcp",
|
||||
f"localhost:{self.config.rpc_port}",
|
||||
"--no-receive-stdout",
|
||||
]
|
||||
)
|
||||
|
||||
logger.info(f"Starting signal-cli daemon: {' '.join(cmd)}")
|
||||
try:
|
||||
self._daemon_proc = subprocess.Popen(
|
||||
cmd,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
except FileNotFoundError:
|
||||
raise ChannelError(
|
||||
f"signal-cli not found at '{self.config.cli_path}'. "
|
||||
"Install: https://github.com/AsamK/signal-cli"
|
||||
) from None
|
||||
|
||||
# Wait for daemon to be ready
|
||||
for _ in range(30):
|
||||
await asyncio.sleep(1)
|
||||
try:
|
||||
_reader, writer = await asyncio.open_connection(
|
||||
"localhost",
|
||||
self.config.rpc_port,
|
||||
)
|
||||
writer.close()
|
||||
await writer.wait_closed()
|
||||
logger.info("signal-cli daemon started")
|
||||
return
|
||||
except (ConnectionRefusedError, OSError):
|
||||
continue
|
||||
|
||||
raise ChannelError("signal-cli daemon failed to start within 30s")
|
||||
|
||||
async def _connect(self) -> None:
|
||||
"""Connect to signal-cli JSON RPC socket."""
|
||||
try:
|
||||
self._reader, self._writer = await asyncio.open_connection(
|
||||
"localhost",
|
||||
self.config.rpc_port,
|
||||
)
|
||||
except Exception as e:
|
||||
raise ChannelError(f"Cannot connect to signal-cli: {e}") from e
|
||||
|
||||
async def _listen_loop(self) -> None:
|
||||
"""Listen for incoming JSON RPC notifications and responses."""
|
||||
while self._running and self._reader:
|
||||
try:
|
||||
line = await self._reader.readline()
|
||||
if not line:
|
||||
break
|
||||
data = json.loads(line.decode())
|
||||
# Dispatch RPC response if it has an 'id' matching a pending call
|
||||
rpc_id = data.get("id")
|
||||
if rpc_id is not None and rpc_id in self._pending_rpcs:
|
||||
fut = self._pending_rpcs.pop(rpc_id)
|
||||
if not fut.done():
|
||||
if "error" in data:
|
||||
fut.set_exception(
|
||||
RuntimeError(f"signal-cli RPC error: {data['error']}")
|
||||
)
|
||||
else:
|
||||
fut.set_result(data.get("result"))
|
||||
continue
|
||||
await self._handle_rpc(data)
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.error(f"Signal listen error: {e}")
|
||||
# Reconnect
|
||||
if self._running:
|
||||
await asyncio.sleep(2)
|
||||
try:
|
||||
await self._connect()
|
||||
except Exception:
|
||||
logger.warning("Signal reconnect failed, exiting listen loop")
|
||||
break
|
||||
|
||||
async def _handle_rpc(self, data: dict) -> None:
|
||||
"""Handle a JSON RPC message from signal-cli."""
|
||||
method = data.get("method", "")
|
||||
|
||||
if method != "receive":
|
||||
return
|
||||
|
||||
params = data.get("params", {})
|
||||
envelope = params.get("envelope", {})
|
||||
source = envelope.get("source") or envelope.get("sourceUuid") or ""
|
||||
source_number = envelope.get("sourceNumber") or source
|
||||
source_name = envelope.get("sourceName") or ""
|
||||
timestamp = envelope.get("timestamp", 0)
|
||||
|
||||
# Ignore messages from self
|
||||
if (
|
||||
source_number == self.config.phone_number
|
||||
or source == self.config.phone_number
|
||||
):
|
||||
logger.debug("Ignoring message from self")
|
||||
return
|
||||
|
||||
# Data message (text)
|
||||
data_msg = envelope.get("dataMessage", {})
|
||||
if data_msg:
|
||||
text = data_msg.get("message", "")
|
||||
group_info = data_msg.get("groupInfo", {})
|
||||
is_group = bool(group_info)
|
||||
chat_id = (
|
||||
group_info.get("groupId", source_number) if is_group else source_number
|
||||
)
|
||||
msg_ts = data_msg.get("timestamp", timestamp)
|
||||
|
||||
media_paths: list[str] = []
|
||||
annotations: list[str] = []
|
||||
_VOICE_TYPES = {
|
||||
"audio/aac",
|
||||
"audio/ogg",
|
||||
"audio/mp4",
|
||||
"audio/mpeg",
|
||||
"audio/opus",
|
||||
}
|
||||
attachments = data_msg.get("attachments", [])
|
||||
for att in attachments:
|
||||
att_size = att.get("size", 0)
|
||||
att_name = att.get("filename", "attachment")
|
||||
att_file = att.get("file") # signal-cli provides local path
|
||||
content_type = att.get("contentType", "")
|
||||
is_voice = content_type in _VOICE_TYPES or att.get("voiceNote", False)
|
||||
media_label = "voice" if is_voice else "attachment"
|
||||
if att_file:
|
||||
from pathlib import Path as _Path
|
||||
|
||||
att_path = _Path(att_file)
|
||||
if att_path.exists():
|
||||
from ..base import MAX_ATTACHMENT_BYTES
|
||||
|
||||
if att_path.stat().st_size > MAX_ATTACHMENT_BYTES:
|
||||
annotations.append(
|
||||
f"[{media_label}: {att_name} - too large ({att_path.stat().st_size} bytes)]"
|
||||
)
|
||||
else:
|
||||
local = self._media_path(f"signal_{att_name}")
|
||||
import shutil
|
||||
|
||||
shutil.copy2(str(att_path), str(local))
|
||||
media_paths.append(str(local))
|
||||
annotations.append(f"[{media_label}: {local}]")
|
||||
else:
|
||||
annotations.append(
|
||||
f"[{media_label}: {att_name} - file not found]"
|
||||
)
|
||||
elif att_size:
|
||||
too_large = self._check_attachment_size(att_size, att_name)
|
||||
if too_large:
|
||||
annotations.append(too_large)
|
||||
else:
|
||||
annotations.append(f"[{media_label}: {att_name}]")
|
||||
|
||||
if not text and not media_paths and not annotations:
|
||||
if not attachments:
|
||||
return
|
||||
# Had attachments but none downloaded successfully
|
||||
if not annotations:
|
||||
text = "[attachment]"
|
||||
|
||||
try:
|
||||
ts = datetime.fromtimestamp(msg_ts / 1000) if msg_ts else datetime.now()
|
||||
except (ValueError, TypeError, OSError):
|
||||
ts = datetime.now()
|
||||
|
||||
was_mentioned = not is_group # DMs always pass
|
||||
if is_group:
|
||||
mentions = data_msg.get("mentions", [])
|
||||
for m in mentions:
|
||||
if (
|
||||
m.get("uuid") == self.config.phone_number
|
||||
or m.get("number") == self.config.phone_number
|
||||
):
|
||||
was_mentioned = True
|
||||
break
|
||||
|
||||
# Cache message_id → sender for reaction targetAuthor
|
||||
self._cache_msg_sender(str(msg_ts), source_number)
|
||||
|
||||
logger.info(
|
||||
"Signal message from %s: %s",
|
||||
source_number,
|
||||
text[:50] if text else "[media]",
|
||||
)
|
||||
await self._enqueue_raw(
|
||||
RawIncoming(
|
||||
sender_id=source_number,
|
||||
chat_id=chat_id,
|
||||
text=text,
|
||||
content_annotations=annotations,
|
||||
media_files=media_paths,
|
||||
timestamp=ts,
|
||||
message_id=str(msg_ts),
|
||||
is_group=is_group,
|
||||
was_mentioned=was_mentioned,
|
||||
metadata={
|
||||
"chat_id": chat_id,
|
||||
"source_name": source_name,
|
||||
"sender_id": source_number,
|
||||
"backend": "signal",
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
# ── Typing indicator ────────────────────────────────────────────
|
||||
|
||||
async def _send_typing_action(self, chat_id: str) -> None:
|
||||
"""Send typing indicator via signal-cli JSON RPC."""
|
||||
params: dict[str, Any] = {
|
||||
"account": self.config.phone_number,
|
||||
}
|
||||
if self._is_group_id(chat_id):
|
||||
params["groupId"] = chat_id
|
||||
else:
|
||||
params["recipient"] = [chat_id]
|
||||
try:
|
||||
await self._rpc_call("sendTyping", params)
|
||||
except Exception:
|
||||
pass # typing indicator is best-effort
|
||||
|
||||
# ── ACK reaction ─────────────────────────────────────────────
|
||||
|
||||
def _cache_msg_sender(self, message_id: str, sender: str) -> None:
|
||||
"""Store message_id → sender mapping for reaction targetAuthor."""
|
||||
if len(self._msg_senders) >= 200:
|
||||
oldest = self._msg_senders_order.popleft()
|
||||
self._msg_senders.pop(oldest, None)
|
||||
self._msg_senders[message_id] = sender
|
||||
self._msg_senders_order.append(message_id)
|
||||
|
||||
async def _send_ack_reaction(
|
||||
self, chat_id: str, message_id: str, emoji: str = "👀"
|
||||
) -> None:
|
||||
"""Send an acknowledgment reaction via signal-cli sendReaction."""
|
||||
target_author = self._msg_senders.get(message_id, "")
|
||||
if not target_author:
|
||||
return # cannot send reaction without knowing the original sender
|
||||
try:
|
||||
params: dict[str, Any] = {
|
||||
"account": self.config.phone_number,
|
||||
"emoji": emoji,
|
||||
"targetAuthor": target_author,
|
||||
"targetTimestamp": int(message_id),
|
||||
}
|
||||
if self._is_group_id(chat_id):
|
||||
params["groupId"] = chat_id
|
||||
else:
|
||||
params["recipient"] = [chat_id]
|
||||
await self._rpc_call("sendReaction", params)
|
||||
except Exception as e:
|
||||
logger.debug(f"Signal ack reaction failed: {e}")
|
||||
|
||||
async def _remove_ack_reaction(
|
||||
self, chat_id: str, message_id: str, emoji: str = "👀"
|
||||
) -> None:
|
||||
"""Remove ACK reaction via signal-cli sendReaction --remove."""
|
||||
target_author = self._msg_senders.get(message_id, "")
|
||||
if not target_author:
|
||||
return
|
||||
try:
|
||||
params: dict[str, Any] = {
|
||||
"account": self.config.phone_number,
|
||||
"emoji": emoji,
|
||||
"targetAuthor": target_author,
|
||||
"targetTimestamp": int(message_id),
|
||||
"remove": True,
|
||||
}
|
||||
if self._is_group_id(chat_id):
|
||||
params["groupId"] = chat_id
|
||||
else:
|
||||
params["recipient"] = [chat_id]
|
||||
await self._rpc_call("sendReaction", params)
|
||||
except Exception as e:
|
||||
logger.debug(f"Signal remove ACK reaction failed: {e}")
|
||||
|
||||
# ── Send ──────────────────────────────────────────────────────
|
||||
|
||||
@staticmethod
|
||||
def _is_group_id(chat_id: str) -> bool:
|
||||
"""Return True if *chat_id* looks like a Signal group ID.
|
||||
|
||||
Group IDs are base64-encoded strings (e.g. ``"aB3d...=="``).
|
||||
Individual recipients are either phone numbers (``"+1234..."``)
|
||||
or UUIDs (``"817ab5e9-..."``) — neither of which is a group.
|
||||
"""
|
||||
return not chat_id.startswith("+") and "-" not in chat_id
|
||||
|
||||
def _is_ready(self) -> bool:
|
||||
return self._writer is not None and not self._writer.is_closing()
|
||||
|
||||
async def _rpc_call(
|
||||
self, method: str, params: dict, timeout: float = 10.0
|
||||
) -> dict | None:
|
||||
"""Send a JSON RPC call to signal-cli and wait for the response."""
|
||||
if not self._writer:
|
||||
return None
|
||||
|
||||
self._rpc_id += 1
|
||||
rpc_id = self._rpc_id
|
||||
request = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": rpc_id,
|
||||
"method": method,
|
||||
"params": params,
|
||||
}
|
||||
|
||||
# Register a Future before sending so the listen loop can resolve it
|
||||
loop = asyncio.get_running_loop()
|
||||
fut: asyncio.Future = loop.create_future()
|
||||
self._pending_rpcs[rpc_id] = fut
|
||||
|
||||
line = json.dumps(request) + "\n"
|
||||
self._writer.write(line.encode())
|
||||
await self._writer.drain()
|
||||
|
||||
try:
|
||||
return await asyncio.wait_for(fut, timeout=timeout)
|
||||
except TimeoutError:
|
||||
self._pending_rpcs.pop(rpc_id, None)
|
||||
logger.warning(f"Signal RPC '{method}' timed out after {timeout}s")
|
||||
return None
|
||||
|
||||
async def _send_chunk(
|
||||
self,
|
||||
chat_id,
|
||||
formatted_text,
|
||||
raw_text,
|
||||
reply_to,
|
||||
metadata,
|
||||
):
|
||||
# Determine if group or individual
|
||||
params: dict[str, Any] = {
|
||||
"message": raw_text,
|
||||
"account": self.config.phone_number,
|
||||
}
|
||||
|
||||
if self._is_group_id(chat_id):
|
||||
params["groupId"] = chat_id
|
||||
else:
|
||||
params["recipient"] = [chat_id]
|
||||
|
||||
await self._rpc_call("send", params)
|
||||
|
||||
# ── Mention stripping ────────────────────────────────────────────
|
||||
|
||||
def _strip_mention(self, text: str) -> str:
|
||||
"""Strip bot mention from Signal messages.
|
||||
|
||||
Signal mentions are embedded as special objects that reference
|
||||
the phone number. The text contains a placeholder character (U+FFFC)
|
||||
at the mention position.
|
||||
"""
|
||||
phone = self.config.phone_number
|
||||
if phone:
|
||||
# Remove phone number if directly mentioned as text
|
||||
text = re.sub(rf"@?{re.escape(phone)}\s*", "", text).strip()
|
||||
# Remove Unicode Object Replacement Character used as mention placeholder
|
||||
text = text.replace("\ufffc", "").strip()
|
||||
return text
|
||||
|
||||
# ── Media send ────────────────────────────────────────────────
|
||||
|
||||
async def _send_media_impl(
|
||||
self,
|
||||
recipient: str,
|
||||
file_path: str,
|
||||
caption: str = "",
|
||||
metadata: dict | None = None,
|
||||
) -> bool:
|
||||
"""Send a media file via signal-cli JSON RPC.
|
||||
|
||||
Uses the "send" RPC method with the attachments parameter.
|
||||
"""
|
||||
chat_id = self._resolve_media_chat_id(recipient, metadata)
|
||||
params: dict[str, Any] = {
|
||||
"account": self.config.phone_number,
|
||||
"attachments": [file_path],
|
||||
}
|
||||
if caption:
|
||||
params["message"] = caption
|
||||
|
||||
if self._is_group_id(chat_id):
|
||||
params["groupId"] = chat_id
|
||||
else:
|
||||
params["recipient"] = [chat_id]
|
||||
|
||||
await self._rpc_call("send", params)
|
||||
return True
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Signal credential validation."""
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def validate_signal(
|
||||
phone_number: str,
|
||||
cli_path: str = "signal-cli",
|
||||
rpc_port: int = 7583,
|
||||
) -> tuple[bool, str]:
|
||||
"""Validate Signal setup by checking signal-cli availability.
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, message).
|
||||
"""
|
||||
import asyncio
|
||||
import subprocess
|
||||
|
||||
if not phone_number:
|
||||
return False, "phone_number is required"
|
||||
|
||||
# Check signal-cli binary
|
||||
loop = asyncio.get_event_loop()
|
||||
|
||||
def _check():
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[cli_path, "--version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return True, f"signal-cli {result.stdout.strip()}"
|
||||
return False, "signal-cli returned error"
|
||||
except FileNotFoundError:
|
||||
return False, f"signal-cli not found at '{cli_path}'"
|
||||
except Exception as e:
|
||||
return False, f"Error: {e}"
|
||||
|
||||
return await loop.run_in_executor(None, _check)
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Signal channel server.
|
||||
|
||||
Standalone script to run the Signal channel with CLI options.
|
||||
|
||||
Usage:
|
||||
python -m EvoScientist.channels.signal.serve --phone-number NUMBER [OPTIONS]
|
||||
|
||||
Examples:
|
||||
# Basic usage
|
||||
python -m EvoScientist.channels.signal.serve --phone-number +1234567890
|
||||
|
||||
# With custom signal-cli path and allowed senders
|
||||
python -m EvoScientist.channels.signal.serve --phone-number +1234567890 --cli-path /usr/local/bin/signal-cli --allow +9876543210
|
||||
|
||||
# With agent and thinking
|
||||
python -m EvoScientist.channels.signal.serve --phone-number +1234567890 --agent --thinking
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
from ...logging_config import configure_logging_from_settings
|
||||
from ..bus import MessageBus
|
||||
from ..standalone import run_standalone
|
||||
from .channel import SignalChannel, SignalConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Signal channel server",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--phone-number",
|
||||
required=True,
|
||||
help="Signal phone number (e.g. +1234567890)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cli-path",
|
||||
default="signal-cli",
|
||||
help="Path to signal-cli binary (default: signal-cli)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--config-dir",
|
||||
help="signal-cli config directory",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--rpc-port",
|
||||
type=int,
|
||||
default=7583,
|
||||
help="signal-cli JSON RPC port (default: 7583)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow",
|
||||
action="append",
|
||||
dest="allowed_senders",
|
||||
help="Allowed sender (phone number). Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--agent",
|
||||
action="store_true",
|
||||
help="Use EvoScientist agent as handler (default: echo)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--thinking",
|
||||
action="store_true",
|
||||
help="Send thinking content as intermediate messages (requires --agent)",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
"""Entry point."""
|
||||
configure_logging_from_settings(default_level=logging.INFO)
|
||||
args = parse_args()
|
||||
|
||||
config = SignalConfig(
|
||||
phone_number=args.phone_number,
|
||||
cli_path=args.cli_path,
|
||||
config_dir=args.config_dir,
|
||||
rpc_port=args.rpc_port,
|
||||
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
|
||||
)
|
||||
|
||||
send_thinking = args.thinking and args.agent
|
||||
bus = MessageBus()
|
||||
channel = SignalChannel(config)
|
||||
|
||||
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,22 @@
|
||||
from ..channel_manager import _parse_csv, register_channel
|
||||
from .channel import SlackChannel, SlackConfig
|
||||
|
||||
__all__ = ["SlackChannel", "SlackConfig"]
|
||||
|
||||
|
||||
def create_from_config(config) -> SlackChannel:
|
||||
allowed = _parse_csv(config.slack_allowed_senders)
|
||||
channels = _parse_csv(config.slack_allowed_channels)
|
||||
proxy = config.slack_proxy or None
|
||||
return SlackChannel(
|
||||
SlackConfig(
|
||||
bot_token=config.slack_bot_token,
|
||||
app_token=config.slack_app_token,
|
||||
allowed_senders=allowed,
|
||||
allowed_channels=channels,
|
||||
proxy=proxy,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
register_channel("slack", create_from_config)
|
||||
@@ -0,0 +1,302 @@
|
||||
"""Slack channel implementation using slack-sdk Socket Mode."""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
|
||||
from ..base import Channel, ChannelError, RawIncoming
|
||||
from ..capabilities import SLACK as SLACK_CAPS
|
||||
from ..config import BaseChannelConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class SlackConfig(BaseChannelConfig):
|
||||
bot_token: str = ""
|
||||
app_token: str = ""
|
||||
text_chunk_limit: int = 4096
|
||||
|
||||
|
||||
class SlackChannel(Channel):
|
||||
"""Slack channel using slack-sdk Socket Mode."""
|
||||
|
||||
name = "slack"
|
||||
|
||||
capabilities = SLACK_CAPS
|
||||
_ready_attrs = ("_web_client",)
|
||||
_mention_pattern = r"<@{bot_id}>\s*"
|
||||
|
||||
def __init__(self, config: SlackConfig):
|
||||
super().__init__(config)
|
||||
self._socket_client = None
|
||||
self._web_client = None
|
||||
self._typing_message_ts: dict[str, str] = {}
|
||||
|
||||
async def start(self) -> None:
|
||||
if not self.config.bot_token:
|
||||
raise ChannelError("Slack bot token is required")
|
||||
if not self.config.app_token:
|
||||
raise ChannelError(
|
||||
"Slack app token is required for Socket Mode (starts with xapp-)"
|
||||
)
|
||||
|
||||
try:
|
||||
from slack_sdk.socket_mode.aiohttp import SocketModeClient
|
||||
from slack_sdk.socket_mode.request import SocketModeRequest
|
||||
from slack_sdk.socket_mode.response import SocketModeResponse
|
||||
from slack_sdk.web.async_client import AsyncWebClient
|
||||
except ImportError:
|
||||
raise ChannelError(
|
||||
"slack-sdk or aiohttp not installed. "
|
||||
"Install with: pip install evoscientist[slack]"
|
||||
) from None
|
||||
|
||||
self._web_client = AsyncWebClient(
|
||||
token=self.config.bot_token,
|
||||
proxy=self._get_proxy(),
|
||||
)
|
||||
|
||||
# Get bot user ID for filtering own messages
|
||||
try:
|
||||
auth = await asyncio.wait_for(
|
||||
self._web_client.auth_test(),
|
||||
timeout=15,
|
||||
)
|
||||
self._bot_user_id = auth["user_id"]
|
||||
except TimeoutError:
|
||||
raise ChannelError(
|
||||
"Slack auth_test timed out — check network and bot token"
|
||||
) from None
|
||||
except Exception as e:
|
||||
raise ChannelError(f"Failed to authenticate Slack bot: {e}") from e
|
||||
|
||||
self._socket_client = SocketModeClient(
|
||||
app_token=self.config.app_token,
|
||||
web_client=self._web_client,
|
||||
)
|
||||
|
||||
async def _event_handler(
|
||||
client: SocketModeClient,
|
||||
req: SocketModeRequest,
|
||||
) -> None:
|
||||
# Acknowledge immediately
|
||||
resp = SocketModeResponse(envelope_id=req.envelope_id)
|
||||
await client.send_socket_mode_response(resp)
|
||||
|
||||
logger.debug(f"Slack socket event: type={req.type}")
|
||||
|
||||
if req.type == "events_api":
|
||||
event = req.payload.get("event", {})
|
||||
event_type = event.get("type", "")
|
||||
if event_type == "message" and "subtype" not in event:
|
||||
is_dm = event.get("channel_type") == "im"
|
||||
await self._on_message(
|
||||
event,
|
||||
is_group=not is_dm,
|
||||
was_mentioned=is_dm,
|
||||
)
|
||||
elif event_type == "app_mention":
|
||||
await self._on_message(
|
||||
event,
|
||||
is_group=True,
|
||||
was_mentioned=True,
|
||||
)
|
||||
|
||||
self._socket_client.socket_mode_request_listeners.append(_event_handler)
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
self._socket_client.connect(),
|
||||
timeout=30,
|
||||
)
|
||||
except TimeoutError:
|
||||
raise ChannelError(
|
||||
"Slack Socket Mode connection timed out — "
|
||||
"check app token (must start with xapp-) and "
|
||||
"ensure Socket Mode is enabled in your Slack app settings"
|
||||
) from None
|
||||
self._running = True
|
||||
logger.info("Slack channel started (Socket Mode)")
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
if self._socket_client:
|
||||
await self._socket_client.close()
|
||||
logger.info("Slack channel stopped")
|
||||
|
||||
# ── Typing indicator (override base) ────────────────────────────
|
||||
|
||||
async def _send_typing_action(self, chat_id: str) -> None:
|
||||
"""Send typing indicator via Slack.
|
||||
|
||||
Slack's Web API and Socket Mode do not expose a dedicated
|
||||
typing-indicator endpoint for bot tokens. We approximate
|
||||
the experience by posting a short-lived status message that
|
||||
is deleted once the real reply is sent (handled by
|
||||
``stop_typing``). When the status post fails we silently
|
||||
fall back to no indicator.
|
||||
"""
|
||||
if not self._web_client:
|
||||
return
|
||||
try:
|
||||
resp = await self._web_client.chat_postMessage(
|
||||
channel=chat_id,
|
||||
text="\u2026", # "…" ellipsis as minimal typing hint
|
||||
)
|
||||
ts = resp.get("ts")
|
||||
if ts:
|
||||
self._typing_message_ts[chat_id] = ts
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def stop_typing(self, chat_id: str) -> None:
|
||||
"""Cancel typing loop and clean up the status message."""
|
||||
# Delete the ephemeral "…" message if we posted one
|
||||
ts = self._typing_message_ts.pop(chat_id, None)
|
||||
if ts and self._web_client:
|
||||
try:
|
||||
await self._web_client.chat_delete(channel=chat_id, ts=ts)
|
||||
except Exception:
|
||||
pass
|
||||
await super().stop_typing(chat_id)
|
||||
|
||||
# ── Send (template method overrides) ──────────────────────────
|
||||
|
||||
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
|
||||
kwargs = {"channel": chat_id}
|
||||
# Always route to thread if thread_ts is present in metadata,
|
||||
# not just for the first chunk (reply_to is only set for chunk 0).
|
||||
if metadata:
|
||||
thread_ts = metadata.get("thread_ts")
|
||||
if thread_ts:
|
||||
kwargs["thread_ts"] = thread_ts
|
||||
|
||||
async def _send(text):
|
||||
await self._web_client.chat_postMessage(text=text, **kwargs)
|
||||
|
||||
await self._send_with_format_fallback(_send, formatted_text, raw_text)
|
||||
|
||||
async def _send_media_impl(
|
||||
self,
|
||||
recipient: str,
|
||||
file_path: str,
|
||||
caption: str = "",
|
||||
metadata: dict | None = None,
|
||||
) -> bool:
|
||||
"""Send a media file through Slack."""
|
||||
channel_id = self._resolve_media_chat_id(recipient, metadata)
|
||||
await self._web_client.files_upload_v2(
|
||||
channel=channel_id,
|
||||
file=file_path,
|
||||
initial_comment=caption or None,
|
||||
)
|
||||
return True
|
||||
|
||||
def _get_bot_identifier(self) -> str | None:
|
||||
return getattr(self, "_bot_user_id", None)
|
||||
|
||||
# ── ACK Reactions ───────────────────────────────────────────────
|
||||
|
||||
async def _send_ack_reaction(
|
||||
self, chat_id: str, message_id: str, emoji: str = "eyes"
|
||||
) -> None:
|
||||
"""Add an emoji reaction to acknowledge receipt."""
|
||||
if self._web_client and message_id:
|
||||
try:
|
||||
await self._web_client.reactions_add(
|
||||
channel=chat_id,
|
||||
timestamp=message_id,
|
||||
name=emoji,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug(f"Slack ACK reaction failed: {e}")
|
||||
|
||||
async def _remove_ack_reaction(
|
||||
self, chat_id: str, message_id: str, emoji: str = "eyes"
|
||||
) -> None:
|
||||
"""Remove the ACK reaction after replying."""
|
||||
if self._web_client and message_id:
|
||||
try:
|
||||
await self._web_client.reactions_remove(
|
||||
channel=chat_id,
|
||||
timestamp=message_id,
|
||||
name=emoji,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug(f"Slack remove ACK reaction failed: {e}")
|
||||
|
||||
async def _on_message(
|
||||
self,
|
||||
event: dict,
|
||||
*,
|
||||
is_group: bool = False,
|
||||
was_mentioned: bool = True,
|
||||
) -> None:
|
||||
"""Handle an incoming Slack message event."""
|
||||
user_id = event.get("user", "")
|
||||
|
||||
# Skip bot's own messages
|
||||
if user_id == getattr(self, "_bot_user_id", None):
|
||||
logger.debug("Skipping own bot message")
|
||||
return
|
||||
|
||||
# Skip bot messages (e.g. from other bots)
|
||||
if event.get("bot_id"):
|
||||
logger.debug(f"Skipping bot message from bot_id={event.get('bot_id')}")
|
||||
return
|
||||
|
||||
channel_id = event.get("channel", "")
|
||||
|
||||
text = event.get("text", "")
|
||||
|
||||
annotations: list[str] = []
|
||||
media_paths: list[str] = []
|
||||
|
||||
# Handle file attachments
|
||||
if self.config.include_attachments:
|
||||
files = event.get("files", [])
|
||||
for file_info in files:
|
||||
file_size = file_info.get("size", 0)
|
||||
filename = file_info.get("name", "unknown")
|
||||
|
||||
url = file_info.get("url_private_download") or file_info.get(
|
||||
"url_private"
|
||||
)
|
||||
if url and self._web_client:
|
||||
headers = {"Authorization": f"Bearer {self.config.bot_token}"}
|
||||
local_path, annotation = await self._download_attachment(
|
||||
url,
|
||||
f"{file_info.get('id', 'unknown')}_{filename}",
|
||||
headers=headers,
|
||||
file_size=file_size,
|
||||
)
|
||||
if local_path:
|
||||
media_paths.append(local_path)
|
||||
if annotation:
|
||||
annotations.append(annotation)
|
||||
|
||||
ts = event.get("ts", "")
|
||||
thread_ts = event.get("thread_ts") or ts
|
||||
try:
|
||||
timestamp = datetime.fromtimestamp(float(ts)) if ts else datetime.now()
|
||||
except (ValueError, TypeError):
|
||||
timestamp = datetime.now()
|
||||
|
||||
await self._enqueue_raw(
|
||||
RawIncoming(
|
||||
sender_id=user_id,
|
||||
chat_id=channel_id,
|
||||
text=text,
|
||||
media_files=media_paths,
|
||||
content_annotations=annotations,
|
||||
timestamp=timestamp,
|
||||
message_id=ts,
|
||||
metadata={"chat_id": channel_id, "thread_ts": thread_ts},
|
||||
is_group=is_group,
|
||||
was_mentioned=was_mentioned,
|
||||
)
|
||||
)
|
||||
logger.info(
|
||||
f"Slack message queued: sender={user_id}, "
|
||||
f"channel={channel_id}, content={text[:50]}"
|
||||
)
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Slack bot token validation."""
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def validate_slack_tokens(
|
||||
bot_token: str,
|
||||
app_token: str | None = None,
|
||||
proxy: str | None = None,
|
||||
) -> tuple[bool, str]:
|
||||
"""Validate Slack bot token via the auth.test API.
|
||||
|
||||
Optionally checks the app-level token format (must start with ``xapp-``).
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, message).
|
||||
"""
|
||||
if not bot_token:
|
||||
return False, "No bot token provided"
|
||||
|
||||
try:
|
||||
import httpx
|
||||
except ImportError:
|
||||
return False, "httpx not installed"
|
||||
|
||||
# Validate bot token via auth.test
|
||||
url = "https://slack.com/api/auth.test"
|
||||
headers = {"Authorization": f"Bearer {bot_token}"}
|
||||
try:
|
||||
async with httpx.AsyncClient(proxy=proxy) as client:
|
||||
resp = await client.post(url, headers=headers, timeout=10)
|
||||
data = resp.json()
|
||||
if not data.get("ok"):
|
||||
error = data.get("error", "unknown error")
|
||||
return False, f"Invalid bot token: {error}"
|
||||
bot_name = data.get("user", "unknown")
|
||||
team = data.get("team", "unknown")
|
||||
except Exception as e:
|
||||
return False, f"Error: {e}"
|
||||
|
||||
# Optionally validate app token format
|
||||
if app_token:
|
||||
if not app_token.startswith("xapp-"):
|
||||
return False, "App token must start with 'xapp-'"
|
||||
|
||||
return True, f"Bot: {bot_name} (team: {team})"
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Slack channel server.
|
||||
|
||||
Standalone script to run the Slack channel with CLI options.
|
||||
|
||||
Usage:
|
||||
python -m EvoScientist.channels.slack.serve --bot-token TOKEN --app-token TOKEN [OPTIONS]
|
||||
|
||||
Examples:
|
||||
# Allow all senders (default)
|
||||
python -m EvoScientist.channels.slack.serve --bot-token xoxb-... --app-token xapp-...
|
||||
|
||||
# Only allow specific senders and channels
|
||||
python -m EvoScientist.channels.slack.serve --bot-token xoxb-... --app-token xapp-... --allow U123 --allow-channel C456
|
||||
|
||||
# With proxy, agent and thinking
|
||||
python -m EvoScientist.channels.slack.serve --bot-token xoxb-... --app-token xapp-... --proxy http://proxy:8080 --agent --thinking
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
from ...logging_config import configure_logging_from_settings
|
||||
from ..bus import MessageBus
|
||||
from ..standalone import run_standalone
|
||||
from .channel import SlackChannel, SlackConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Slack channel server",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--bot-token",
|
||||
required=True,
|
||||
help="Slack bot token (xoxb-...)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--app-token",
|
||||
required=True,
|
||||
help="Slack app-level token for Socket Mode (xapp-...)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow",
|
||||
action="append",
|
||||
dest="allowed_senders",
|
||||
help="Allowed sender (Slack user ID). Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow-channel",
|
||||
action="append",
|
||||
dest="allowed_channels",
|
||||
help="Allowed channel ID. Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--proxy",
|
||||
help="HTTP proxy URL for Slack API requests",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--agent",
|
||||
action="store_true",
|
||||
help="Use EvoScientist agent as handler (default: echo)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--thinking",
|
||||
action="store_true",
|
||||
help="Send thinking content as intermediate messages (requires --agent)",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
"""Entry point."""
|
||||
configure_logging_from_settings(default_level=logging.INFO)
|
||||
args = parse_args()
|
||||
|
||||
config = SlackConfig(
|
||||
bot_token=args.bot_token,
|
||||
app_token=args.app_token,
|
||||
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
|
||||
allowed_channels=set(args.allowed_channels) if args.allowed_channels else None,
|
||||
proxy=args.proxy,
|
||||
)
|
||||
|
||||
send_thinking = args.thinking and args.agent
|
||||
bus = MessageBus()
|
||||
channel = SlackChannel(config)
|
||||
|
||||
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,183 @@
|
||||
"""Shared standalone runner for channel servers.
|
||||
|
||||
Provides the channel-agnostic agent loop that any channel can use to
|
||||
run headless — consuming inbound messages from the bus, streaming
|
||||
agent events, and dispatching outbound replies.
|
||||
|
||||
Usage from a channel's ``main()``::
|
||||
|
||||
from EvoScientist.channels.standalone import run_standalone
|
||||
|
||||
channel = SomeChannel(config)
|
||||
bus = MessageBus()
|
||||
run_standalone(channel, bus, use_agent=True, send_thinking=True)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import signal
|
||||
|
||||
from .base import Channel
|
||||
from .bus import MessageBus
|
||||
from .bus.events import OutboundMessage
|
||||
from .consumer import InboundConsumer
|
||||
from .debug import emit_debug_event
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _channel_trace_enabled(channel: Channel) -> bool:
|
||||
"""Check if debug tracing is enabled on the channel."""
|
||||
try:
|
||||
return channel.is_debug_trace_enabled()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
async def _deliver_outbound(channel: Channel, msg: OutboundMessage) -> None:
|
||||
"""Deliver an outbound message, including any media attachments."""
|
||||
if msg.content:
|
||||
sent = await channel.send(msg)
|
||||
if not sent:
|
||||
raise RuntimeError("send() returned False")
|
||||
for media_path in msg.media:
|
||||
media_ok = await channel.send_media(
|
||||
recipient=msg.chat_id,
|
||||
file_path=media_path,
|
||||
metadata=msg.metadata,
|
||||
)
|
||||
if not media_ok:
|
||||
raise RuntimeError(f"send_media() returned False for {media_path}")
|
||||
|
||||
|
||||
async def standalone_outbound_dispatcher(
|
||||
bus: MessageBus,
|
||||
channel: Channel,
|
||||
) -> None:
|
||||
"""Consume outbound messages from the bus and send via channel."""
|
||||
while True:
|
||||
try:
|
||||
msg: OutboundMessage = await asyncio.wait_for(
|
||||
bus.consume_outbound(),
|
||||
timeout=1.0,
|
||||
)
|
||||
except TimeoutError:
|
||||
continue
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
|
||||
try:
|
||||
await _deliver_outbound(channel, msg)
|
||||
except Exception as e:
|
||||
emit_debug_event(
|
||||
logger,
|
||||
"standalone_dispatch_error",
|
||||
channel=channel.name,
|
||||
enabled=_channel_trace_enabled(channel),
|
||||
recipient=msg.recipient,
|
||||
error=str(e),
|
||||
)
|
||||
logger.error(f"Error sending outbound: {e}")
|
||||
|
||||
|
||||
async def _async_main(
|
||||
channel: Channel,
|
||||
bus: MessageBus,
|
||||
use_agent: bool,
|
||||
send_thinking: bool,
|
||||
) -> None:
|
||||
"""Async entry point — gather channel, dispatcher and optional consumer."""
|
||||
from .channel_manager import ChannelManager
|
||||
|
||||
channel.set_bus(bus)
|
||||
if send_thinking:
|
||||
channel.send_thinking = True
|
||||
|
||||
# Create a lightweight manager for the consumer to use
|
||||
manager = ChannelManager(bus)
|
||||
manager._channels[channel.name] = channel
|
||||
|
||||
await manager.start_health()
|
||||
|
||||
tasks = [channel.run()]
|
||||
|
||||
dispatcher = standalone_outbound_dispatcher(bus, channel)
|
||||
tasks.append(dispatcher)
|
||||
|
||||
consumer: InboundConsumer | None = None
|
||||
if use_agent:
|
||||
logger.info("Loading EvoScientist agent...")
|
||||
from ..EvoScientist import create_cli_agent
|
||||
|
||||
agent = create_cli_agent()
|
||||
logger.info("Agent loaded")
|
||||
|
||||
consumer = InboundConsumer(
|
||||
bus=bus,
|
||||
manager=manager,
|
||||
agent=agent,
|
||||
thread_id="",
|
||||
send_thinking=send_thinking,
|
||||
)
|
||||
manager.register_health_provider("consumer", lambda: consumer.metrics)
|
||||
tasks.append(consumer.run())
|
||||
if send_thinking:
|
||||
logger.info("Thinking messages enabled")
|
||||
|
||||
async def _graceful_shutdown() -> None:
|
||||
"""Graceful shutdown: drain consumer, flush outbound, stop channel."""
|
||||
logger.info("Graceful shutdown initiated...")
|
||||
if consumer is not None:
|
||||
await consumer.stop()
|
||||
# Drain outbound queue before stopping the channel
|
||||
drained = 0
|
||||
while True:
|
||||
try:
|
||||
msg = bus.outbound.get_nowait()
|
||||
except asyncio.QueueEmpty:
|
||||
break
|
||||
try:
|
||||
await asyncio.wait_for(_deliver_outbound(channel, msg), timeout=5.0)
|
||||
if msg.content or msg.media:
|
||||
drained += 1
|
||||
except Exception:
|
||||
pass
|
||||
if drained:
|
||||
logger.info(f"Outbound drain: {drained} sent")
|
||||
channel._running = False
|
||||
await channel.stop()
|
||||
await manager.stop_health()
|
||||
|
||||
loop = asyncio.get_event_loop()
|
||||
for sig in (signal.SIGINT, signal.SIGTERM):
|
||||
loop.add_signal_handler(
|
||||
sig,
|
||||
lambda s=sig: asyncio.create_task(_graceful_shutdown()),
|
||||
)
|
||||
|
||||
await asyncio.gather(*tasks)
|
||||
|
||||
|
||||
def run_standalone(
|
||||
channel: Channel,
|
||||
bus: MessageBus,
|
||||
*,
|
||||
use_agent: bool = False,
|
||||
send_thinking: bool = False,
|
||||
) -> None:
|
||||
"""Synchronous entry point that spins up the standalone runner.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
channel:
|
||||
A fully-configured :class:`Channel` instance.
|
||||
bus:
|
||||
The :class:`MessageBus` shared with *channel*.
|
||||
use_agent:
|
||||
When ``True``, load the EvoScientist agent and process inbound
|
||||
messages through it.
|
||||
send_thinking:
|
||||
When ``True`` **and** *use_agent* is set, forward intermediate
|
||||
thinking messages to the channel.
|
||||
"""
|
||||
asyncio.run(_async_main(channel, bus, use_agent, send_thinking))
|
||||
@@ -0,0 +1,19 @@
|
||||
from ..channel_manager import _parse_csv, register_channel
|
||||
from .channel import TelegramChannel, TelegramConfig
|
||||
|
||||
__all__ = ["TelegramChannel", "TelegramConfig"]
|
||||
|
||||
|
||||
def create_from_config(config) -> TelegramChannel:
|
||||
allowed = _parse_csv(config.telegram_allowed_senders)
|
||||
proxy = config.telegram_proxy or None
|
||||
return TelegramChannel(
|
||||
TelegramConfig(
|
||||
bot_token=config.telegram_bot_token,
|
||||
allowed_senders=allowed,
|
||||
proxy=proxy,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
register_channel("telegram", create_from_config)
|
||||
@@ -0,0 +1,315 @@
|
||||
"""Telegram channel implementation using python-telegram-bot."""
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import ClassVar
|
||||
|
||||
from ..base import (
|
||||
AUDIO_EXTS,
|
||||
IMAGE_EXTS,
|
||||
VIDEO_EXTS,
|
||||
Channel,
|
||||
ChannelError,
|
||||
RawIncoming,
|
||||
)
|
||||
from ..capabilities import TELEGRAM as TELEGRAM_CAPS
|
||||
from ..config import BaseChannelConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class TelegramConfig(BaseChannelConfig):
|
||||
bot_token: str = ""
|
||||
text_chunk_limit: int = 4096
|
||||
|
||||
|
||||
class TelegramChannel(Channel):
|
||||
"""Telegram channel using python-telegram-bot with long polling."""
|
||||
|
||||
name = "telegram"
|
||||
|
||||
capabilities = TELEGRAM_CAPS
|
||||
_typing_interval: float = 4.0
|
||||
_ready_attrs = ("_app",)
|
||||
_non_retryable_patterns = ("parse", "can't parse")
|
||||
_mention_pattern = r"(?i)@{bot_id}\s*"
|
||||
|
||||
def __init__(self, config: TelegramConfig):
|
||||
super().__init__(config)
|
||||
self._app = None
|
||||
self._bot_username: str = ""
|
||||
|
||||
async def start(self) -> None:
|
||||
if not self.config.bot_token:
|
||||
raise ChannelError("Telegram bot token is required")
|
||||
|
||||
try:
|
||||
from telegram.ext import (
|
||||
ApplicationBuilder,
|
||||
MessageHandler,
|
||||
filters,
|
||||
)
|
||||
except ImportError:
|
||||
raise ChannelError(
|
||||
"python-telegram-bot not installed. "
|
||||
"Install with: pip install evoscientist[telegram]"
|
||||
) from None
|
||||
|
||||
builder = ApplicationBuilder().token(self.config.bot_token)
|
||||
|
||||
if self.config.proxy:
|
||||
builder = builder.proxy(self.config.proxy).get_updates_proxy(
|
||||
self.config.proxy
|
||||
)
|
||||
self._app = builder.build()
|
||||
|
||||
# Accept text and media message types
|
||||
media_filter = filters.TEXT
|
||||
if self.config.include_attachments:
|
||||
media_filter = (
|
||||
filters.TEXT
|
||||
| filters.PHOTO
|
||||
| filters.VOICE
|
||||
| filters.AUDIO
|
||||
| filters.Document.ALL
|
||||
| filters.VIDEO
|
||||
| filters.Sticker.ALL
|
||||
| filters.LOCATION
|
||||
)
|
||||
|
||||
self._app.add_handler(
|
||||
MessageHandler(media_filter & ~filters.COMMAND, self._on_message)
|
||||
)
|
||||
|
||||
await self._app.initialize()
|
||||
# Cache bot username for @mention detection in groups
|
||||
bot_info = await self._app.bot.get_me()
|
||||
self._bot_username = (bot_info.username or "").lower()
|
||||
await self._app.start()
|
||||
await self._app.updater.start_polling(drop_pending_updates=True)
|
||||
self._running = True
|
||||
logger.info("Telegram channel started (polling)")
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
if self._app:
|
||||
if self._app.updater and self._app.updater.running:
|
||||
await self._app.updater.stop()
|
||||
await self._app.stop()
|
||||
await self._app.shutdown()
|
||||
logger.info("Telegram channel stopped")
|
||||
|
||||
# ── Typing indicator (override base) ────────────────────────────
|
||||
|
||||
async def _send_typing_action(self, chat_id: str) -> None:
|
||||
"""Send typing action via Telegram Bot API."""
|
||||
if self._app:
|
||||
await self._app.bot.send_chat_action(
|
||||
chat_id=int(chat_id),
|
||||
action="typing",
|
||||
)
|
||||
|
||||
# ── Send (template method overrides) ──────────────────────────
|
||||
|
||||
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
|
||||
reply_id = int(reply_to) if reply_to else None
|
||||
|
||||
async def _send(text):
|
||||
await self._app.bot.send_message(
|
||||
chat_id=int(chat_id),
|
||||
text=text,
|
||||
parse_mode="HTML" if text == formatted_text else None,
|
||||
reply_to_message_id=reply_id,
|
||||
)
|
||||
|
||||
await self._send_with_format_fallback(_send, formatted_text, raw_text)
|
||||
|
||||
_MEDIA_SENDERS: ClassVar[dict] = {
|
||||
IMAGE_EXTS: ("send_photo", "photo"),
|
||||
VIDEO_EXTS: ("send_video", "video"),
|
||||
AUDIO_EXTS: ("send_audio", "audio"),
|
||||
}
|
||||
|
||||
async def _send_media_impl(
|
||||
self,
|
||||
recipient: str,
|
||||
file_path: str,
|
||||
caption: str = "",
|
||||
metadata: dict | None = None,
|
||||
) -> bool:
|
||||
"""Send a media file through Telegram."""
|
||||
chat_id = int(self._resolve_media_chat_id(recipient, metadata))
|
||||
cap = caption or None
|
||||
ext = Path(file_path).suffix.lower()
|
||||
for exts, (method, param) in self._MEDIA_SENDERS.items():
|
||||
if ext in exts:
|
||||
await getattr(self._app.bot, method)(
|
||||
chat_id=chat_id,
|
||||
caption=cap,
|
||||
**{param: file_path},
|
||||
)
|
||||
return True
|
||||
await self._app.bot.send_document(
|
||||
chat_id=chat_id,
|
||||
document=file_path,
|
||||
caption=cap,
|
||||
)
|
||||
return True
|
||||
|
||||
def _get_bot_identifier(self) -> str | None:
|
||||
return self._bot_username or None
|
||||
|
||||
async def _send_ack_reaction(
|
||||
self, chat_id: str, message_id: str, emoji: str = "👀"
|
||||
) -> None:
|
||||
"""Send an acknowledgment reaction via Telegram."""
|
||||
if self._app:
|
||||
try:
|
||||
from telegram import ReactionTypeEmoji
|
||||
|
||||
await self._app.bot.set_message_reaction(
|
||||
chat_id=int(chat_id),
|
||||
message_id=int(message_id),
|
||||
reaction=[ReactionTypeEmoji(emoji)],
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug(f"Telegram ACK reaction failed: {e}")
|
||||
|
||||
async def _remove_ack_reaction(
|
||||
self, chat_id: str, message_id: str, emoji: str = "👀"
|
||||
) -> None:
|
||||
"""Remove the ack reaction by setting empty reaction list."""
|
||||
if self._app:
|
||||
try:
|
||||
await self._app.bot.set_message_reaction(
|
||||
chat_id=int(chat_id),
|
||||
message_id=int(message_id),
|
||||
reaction=[],
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug(f"Telegram remove ACK reaction failed: {e}")
|
||||
|
||||
async def _on_message(self, update, context) -> None:
|
||||
"""Handler callback for text, photos, voice, audio, documents, video."""
|
||||
if not update.message:
|
||||
return
|
||||
|
||||
message = update.message
|
||||
user_id = str(message.from_user.id)
|
||||
chat_id = str(message.chat_id)
|
||||
|
||||
# Detect group and mention status for centralized gating
|
||||
is_group = message.chat.type in ("group", "supergroup")
|
||||
was_mentioned = True # DM default
|
||||
if is_group and self._bot_username:
|
||||
text_check = (message.text or message.caption or "").lower()
|
||||
was_mentioned = f"@{self._bot_username}" in text_check
|
||||
|
||||
content_parts: list[str] = []
|
||||
media_paths: list[str] = []
|
||||
|
||||
# Text content
|
||||
if message.text:
|
||||
content_parts.append(message.text)
|
||||
if message.caption:
|
||||
content_parts.append(message.caption)
|
||||
|
||||
# Handle media files
|
||||
annotations: list[str] = []
|
||||
if self.config.include_attachments:
|
||||
media_file = None
|
||||
media_type = None
|
||||
|
||||
if message.photo:
|
||||
media_file = message.photo[-1] # Largest size
|
||||
media_type = "image"
|
||||
elif message.voice:
|
||||
media_file = message.voice
|
||||
media_type = "voice"
|
||||
elif message.audio:
|
||||
media_file = message.audio
|
||||
media_type = "audio"
|
||||
elif message.video:
|
||||
media_file = message.video
|
||||
media_type = "video"
|
||||
elif message.document:
|
||||
media_file = message.document
|
||||
media_type = "file"
|
||||
elif message.sticker:
|
||||
media_file = message.sticker
|
||||
media_type = "sticker"
|
||||
|
||||
# Location is not a downloadable file — handle separately
|
||||
if message.location and not media_file:
|
||||
loc = message.location
|
||||
annotations.append(f"[位置] ({loc.latitude}, {loc.longitude})")
|
||||
|
||||
if media_file and self._app:
|
||||
file_size = getattr(media_file, "file_size", 0) or 0
|
||||
too_large = self._check_attachment_size(file_size, media_type)
|
||||
if too_large:
|
||||
annotations.append(too_large)
|
||||
else:
|
||||
try:
|
||||
file = await self._app.bot.get_file(
|
||||
media_file.file_id,
|
||||
)
|
||||
ext = self._get_extension(
|
||||
media_type,
|
||||
getattr(media_file, "mime_type", None),
|
||||
)
|
||||
file_path = self._media_path(f"{media_file.file_id[:16]}{ext}")
|
||||
await file.download_to_drive(str(file_path))
|
||||
|
||||
media_paths.append(str(file_path))
|
||||
annotations.append(f"[{media_type}: {file_path}]")
|
||||
logger.debug(f"Downloaded {media_type} to {file_path}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to download media: {e}")
|
||||
annotations.append(f"[{media_type}: download failed]")
|
||||
|
||||
text_content = "\n".join(content_parts) if content_parts else ""
|
||||
|
||||
await self._enqueue_raw(
|
||||
RawIncoming(
|
||||
sender_id=user_id,
|
||||
chat_id=chat_id,
|
||||
text=text_content,
|
||||
media_files=media_paths,
|
||||
content_annotations=annotations,
|
||||
timestamp=message.date or datetime.now(),
|
||||
message_id=str(message.message_id),
|
||||
metadata={"chat_id": chat_id},
|
||||
is_group=is_group,
|
||||
was_mentioned=was_mentioned,
|
||||
)
|
||||
)
|
||||
|
||||
_MIME_TO_EXT: ClassVar[dict[str, str]] = {
|
||||
"image/jpeg": ".jpg",
|
||||
"image/png": ".png",
|
||||
"image/gif": ".gif",
|
||||
"image/webp": ".webp",
|
||||
"audio/ogg": ".ogg",
|
||||
"audio/mpeg": ".mp3",
|
||||
"audio/mp4": ".m4a",
|
||||
"video/mp4": ".mp4",
|
||||
"video/quicktime": ".mov",
|
||||
}
|
||||
_TYPE_TO_EXT: ClassVar[dict[str, str]] = {
|
||||
"image": ".jpg",
|
||||
"voice": ".ogg",
|
||||
"audio": ".mp3",
|
||||
"video": ".mp4",
|
||||
"file": "",
|
||||
"sticker": ".webp",
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _get_extension(media_type: str, mime_type: str | None) -> str:
|
||||
"""Get file extension based on media type and MIME type."""
|
||||
if mime_type and mime_type in TelegramChannel._MIME_TO_EXT:
|
||||
return TelegramChannel._MIME_TO_EXT[mime_type]
|
||||
return TelegramChannel._TYPE_TO_EXT.get(media_type, "")
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Telegram bot token validation."""
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def validate_telegram_token(
|
||||
token: str, proxy: str | None = None
|
||||
) -> tuple[bool, str]:
|
||||
"""Validate a Telegram bot token via the getMe API.
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, message).
|
||||
"""
|
||||
if not token:
|
||||
return False, "No token provided"
|
||||
|
||||
try:
|
||||
import httpx
|
||||
except ImportError:
|
||||
return False, "httpx not installed"
|
||||
|
||||
url = f"https://api.telegram.org/bot{token}/getMe"
|
||||
try:
|
||||
async with httpx.AsyncClient(proxy=proxy) as client:
|
||||
resp = await client.get(url, timeout=10)
|
||||
data = resp.json()
|
||||
if data.get("ok"):
|
||||
username = data["result"].get("username", "unknown")
|
||||
return True, f"Bot: @{username}"
|
||||
return False, "Invalid token"
|
||||
except Exception as e:
|
||||
return False, f"Error: {e}"
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Telegram channel server.
|
||||
|
||||
Standalone script to run the Telegram channel with CLI options.
|
||||
|
||||
Usage:
|
||||
python -m EvoScientist.channels.telegram.serve --bot-token TOKEN [OPTIONS]
|
||||
|
||||
Examples:
|
||||
# Allow all senders (default)
|
||||
python -m EvoScientist.channels.telegram.serve --bot-token TOKEN
|
||||
|
||||
# Only allow specific senders
|
||||
python -m EvoScientist.channels.telegram.serve --bot-token TOKEN --allow 123456 --allow 789012
|
||||
|
||||
# With agent and thinking
|
||||
python -m EvoScientist.channels.telegram.serve --bot-token TOKEN --agent --thinking
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
from ...logging_config import configure_logging_from_settings
|
||||
from ..bus import MessageBus
|
||||
from ..standalone import run_standalone
|
||||
from .channel import TelegramChannel, TelegramConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Telegram channel server",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--bot-token",
|
||||
required=True,
|
||||
help="Telegram bot token from @BotFather",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow",
|
||||
action="append",
|
||||
dest="allowed_senders",
|
||||
help="Allowed sender (Telegram user ID). Can be used multiple times.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--agent",
|
||||
action="store_true",
|
||||
help="Use EvoScientist agent as handler (default: echo)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--thinking",
|
||||
action="store_true",
|
||||
help="Send thinking content as intermediate messages (requires --agent)",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
"""Entry point."""
|
||||
configure_logging_from_settings(default_level=logging.INFO)
|
||||
args = parse_args()
|
||||
|
||||
config = TelegramConfig(
|
||||
bot_token=args.bot_token,
|
||||
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
|
||||
)
|
||||
|
||||
send_thinking = args.thinking and args.agent
|
||||
bus = MessageBus()
|
||||
channel = TelegramChannel(config)
|
||||
|
||||
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,69 @@
|
||||
"""WeChat channel implementations for EvoScientist.
|
||||
|
||||
Supports multiple WeChat backends:
|
||||
- **wecom**: 企业微信应用 (WeCom / WeChat Work) via official API
|
||||
— Most stable, pure HTTP, no third-party dependencies
|
||||
- **wechatmp**: 微信公众号 (WeChat Official Account) via official API
|
||||
— Pure HTTP webhook, suitable for public-facing bots
|
||||
|
||||
Both backends use httpx (already a core dependency) and receive messages
|
||||
via HTTP webhook, send replies via REST API.
|
||||
|
||||
Usage in config:
|
||||
channel_enabled = "wechat"
|
||||
wechat_backend = "wecom" # or "wechatmp"
|
||||
|
||||
# WeCom settings
|
||||
wechat_wecom_corp_id = "..."
|
||||
wechat_wecom_agent_id = "..."
|
||||
wechat_wecom_secret = "..."
|
||||
wechat_wecom_token = "..."
|
||||
wechat_wecom_encoding_aes_key = "..."
|
||||
wechat_webhook_port = 9001
|
||||
|
||||
# OR: Official Account settings
|
||||
wechat_mp_app_id = "..."
|
||||
wechat_mp_app_secret = "..."
|
||||
wechat_mp_token = "..."
|
||||
wechat_mp_encoding_aes_key = "..."
|
||||
wechat_webhook_port = 9001
|
||||
"""
|
||||
|
||||
from ..channel_manager import _parse_csv, register_channel
|
||||
from .channel import WeChatChannel, WeChatMPConfig, WeComConfig
|
||||
|
||||
__all__ = ["WeChatChannel", "WeChatMPConfig", "WeComConfig"]
|
||||
|
||||
|
||||
def create_from_config(config) -> WeChatChannel:
|
||||
backend = config.wechat_backend or "wecom"
|
||||
allowed = _parse_csv(config.wechat_allowed_senders)
|
||||
proxy = config.wechat_proxy or None
|
||||
port = int(config.wechat_webhook_port or 9001)
|
||||
|
||||
if backend == "wechatmp":
|
||||
mp_config = WeChatMPConfig(
|
||||
app_id=config.wechat_mp_app_id,
|
||||
app_secret=config.wechat_mp_app_secret,
|
||||
token=config.wechat_mp_token,
|
||||
encoding_aes_key=config.wechat_mp_encoding_aes_key,
|
||||
webhook_port=port,
|
||||
allowed_senders=allowed,
|
||||
proxy=proxy,
|
||||
)
|
||||
return WeChatChannel(mp_config, backend="wechatmp")
|
||||
else:
|
||||
wecom_config = WeComConfig(
|
||||
corp_id=config.wechat_wecom_corp_id,
|
||||
agent_id=config.wechat_wecom_agent_id,
|
||||
secret=config.wechat_wecom_secret,
|
||||
token=config.wechat_wecom_token,
|
||||
encoding_aes_key=config.wechat_wecom_encoding_aes_key,
|
||||
webhook_port=port,
|
||||
allowed_senders=allowed,
|
||||
proxy=proxy,
|
||||
)
|
||||
return WeChatChannel(wecom_config, backend="wecom")
|
||||
|
||||
|
||||
register_channel("wechat", create_from_config)
|
||||
@@ -0,0 +1,906 @@
|
||||
"""WeChat channel implementation.
|
||||
|
||||
Supports two backends via a unified Channel interface:
|
||||
|
||||
1. **wecom** (企业微信应用): Corporate WeChat official API
|
||||
- Receives messages via HTTP callback (XML + optional AES encryption)
|
||||
- Sends replies via REST API (POST /cgi-bin/message/send)
|
||||
- Supports text, image, file, markdown messages
|
||||
- Token auto-refresh with 2-hour TTL
|
||||
|
||||
2. **wechatmp** (微信公众号): WeChat Official Account API
|
||||
- Receives messages via HTTP callback (XML + optional AES encryption)
|
||||
- Sends replies via REST API (POST /cgi-bin/message/custom/send)
|
||||
- Supports text, image, news messages
|
||||
|
||||
Both backends use httpx (already a core dependency) and aiohttp for
|
||||
webhook server — matching the Feishu channel pattern.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from aiohttp import web
|
||||
|
||||
from ..base import Channel, ChannelError, RawIncoming
|
||||
from ..capabilities import WECHAT as WECHAT_CAPS
|
||||
from ..config import BaseChannelConfig
|
||||
from ..mixins import TokenMixin, WebhookMixin
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ── Markdown → plain text (fallback for WeChat text messages) ────
|
||||
|
||||
|
||||
def _strip_markdown(text: str) -> str:
|
||||
"""Strip Markdown formatting for plain-text WeChat messages."""
|
||||
# Remove code blocks
|
||||
text = re.sub(r"```[\s\S]*?```", lambda m: m.group(0).strip("`").strip(), text)
|
||||
# Remove inline code
|
||||
text = re.sub(r"`([^`]+)`", r"\1", text)
|
||||
# Remove bold
|
||||
text = re.sub(r"\*\*(.+?)\*\*", r"\1", text)
|
||||
# Remove italic
|
||||
text = re.sub(r"(?<!\w)_([^_]+?)_(?!\w)", r"\1", text)
|
||||
# Remove strikethrough
|
||||
text = re.sub(r"~~(.+?)~~", r"\1", text)
|
||||
# Convert links
|
||||
text = re.sub(r"\[([^\]]+)\]\(([^)]+)\)", r"\1(\2)", text)
|
||||
# Remove heading markers
|
||||
text = re.sub(r"^#{1,6}\s+", "", text, flags=re.MULTILINE)
|
||||
# Convert list items
|
||||
text = re.sub(r"^[\-\*]\s+", "• ", text, flags=re.MULTILINE)
|
||||
return text
|
||||
|
||||
|
||||
# ── Config dataclasses ───────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class WeComConfig(BaseChannelConfig):
|
||||
"""Configuration for WeCom (企业微信) backend."""
|
||||
|
||||
corp_id: str = ""
|
||||
agent_id: str = ""
|
||||
secret: str = ""
|
||||
token: str = ""
|
||||
encoding_aes_key: str = ""
|
||||
webhook_port: int = 9001
|
||||
|
||||
|
||||
@dataclass
|
||||
class WeChatMPConfig(BaseChannelConfig):
|
||||
"""Configuration for WeChat Official Account (公众号) backend."""
|
||||
|
||||
app_id: str = ""
|
||||
app_secret: str = ""
|
||||
token: str = ""
|
||||
encoding_aes_key: str = ""
|
||||
webhook_port: int = 9001
|
||||
|
||||
|
||||
# ── Unified WeChat Channel ───────────────────────────────────────
|
||||
|
||||
|
||||
class WeChatChannel(Channel, WebhookMixin, TokenMixin):
|
||||
capabilities = WECHAT_CAPS
|
||||
"""Unified WeChat channel supporting WeCom and Official Account backends.
|
||||
|
||||
Architecture follows the same pattern as FeishuChannel:
|
||||
- HTTP webhook server (aiohttp) for inbound messages
|
||||
- REST API calls (httpx) for outbound messages
|
||||
- Token auto-refresh
|
||||
"""
|
||||
|
||||
name = "wechat"
|
||||
_typing_interval: float = 5.0 # WeChat has no typing API, but keep for interface
|
||||
_ready_attrs = ("_http_client", "_access_token")
|
||||
_rate_limit_patterns = ("45009", "frequency", "freq")
|
||||
_rate_limit_delay = 2.0
|
||||
_mention_pattern = r"@\S+\s*"
|
||||
_mention_strip_count = 1
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: WeComConfig | WeChatMPConfig,
|
||||
backend: str = "wecom",
|
||||
):
|
||||
super().__init__(config)
|
||||
self._backend = backend
|
||||
self._access_token: str | None = None
|
||||
self._token_expires: float = 0
|
||||
self._runner = None
|
||||
self._site = None
|
||||
self._http_client = None
|
||||
self._crypto = None # WeChatCrypto instance (optional)
|
||||
self._typing_message_ids: dict[
|
||||
str, list[str]
|
||||
] = {} # chat_id → [msgid, ...] for typing recall
|
||||
self._background_tasks: set[asyncio.Task] = set()
|
||||
|
||||
# ── Lifecycle ─────────────────────────────────────────────────
|
||||
|
||||
def _webhook_routes(self) -> list[tuple[str, str, Any]]:
|
||||
"""Return HTTP routes for the shared webhook server."""
|
||||
return [
|
||||
("GET", "/wechat/callback", self._handle_verify),
|
||||
("POST", "/wechat/callback", self._handle_message),
|
||||
]
|
||||
|
||||
async def start(self) -> None:
|
||||
try:
|
||||
import httpx
|
||||
from aiohttp import web
|
||||
except ImportError:
|
||||
raise ChannelError(
|
||||
"aiohttp or httpx not installed. "
|
||||
"Install with: pip install aiohttp httpx"
|
||||
) from None
|
||||
|
||||
self._validate_config()
|
||||
|
||||
import httpx
|
||||
|
||||
self._http_client = httpx.AsyncClient(
|
||||
timeout=15,
|
||||
proxy=self._get_proxy(),
|
||||
)
|
||||
|
||||
# Set up message encryption if configured
|
||||
if self.config.encoding_aes_key and self.config.token:
|
||||
from .crypto import WeChatCrypto
|
||||
|
||||
app_id = self._get_app_id()
|
||||
self._crypto = WeChatCrypto(
|
||||
token=self.config.token,
|
||||
encoding_aes_key=self.config.encoding_aes_key,
|
||||
app_id=app_id,
|
||||
)
|
||||
|
||||
# Verify credentials by fetching initial token
|
||||
await self._refresh_token()
|
||||
|
||||
if not getattr(self, "_shared_webhook_server", None):
|
||||
app = web.Application()
|
||||
app.router.add_get("/wechat/callback", self._handle_verify)
|
||||
app.router.add_post("/wechat/callback", self._handle_message)
|
||||
|
||||
self._runner = web.AppRunner(app)
|
||||
await self._runner.setup()
|
||||
self._site = web.TCPSite(
|
||||
self._runner,
|
||||
"0.0.0.0",
|
||||
self.config.webhook_port,
|
||||
)
|
||||
await self._site.start()
|
||||
|
||||
self._running = True
|
||||
logger.info(
|
||||
f"WeChat channel started "
|
||||
f"(backend={self._backend}, "
|
||||
f"webhook on port {self.config.webhook_port})"
|
||||
)
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
if self._site:
|
||||
await self._site.stop()
|
||||
if self._runner:
|
||||
await self._runner.cleanup()
|
||||
if self._http_client:
|
||||
await self._http_client.aclose()
|
||||
self._http_client = None
|
||||
self._access_token = None
|
||||
logger.info("WeChat channel stopped")
|
||||
|
||||
def _validate_config(self) -> None:
|
||||
"""Validate required config fields based on backend."""
|
||||
if self._backend == "wecom":
|
||||
cfg = self.config
|
||||
if not cfg.corp_id:
|
||||
raise ChannelError("WeCom corp_id is required")
|
||||
if not cfg.secret:
|
||||
raise ChannelError("WeCom secret is required")
|
||||
if not cfg.agent_id:
|
||||
raise ChannelError("WeCom agent_id is required")
|
||||
elif self._backend == "wechatmp":
|
||||
cfg = self.config
|
||||
if not cfg.app_id:
|
||||
raise ChannelError("WeChat MP app_id is required")
|
||||
if not cfg.app_secret:
|
||||
raise ChannelError("WeChat MP app_secret is required")
|
||||
|
||||
def _get_app_id(self) -> str:
|
||||
"""Return the app identifier for crypto operations."""
|
||||
if self._backend == "wecom":
|
||||
return self.config.corp_id
|
||||
return self.config.app_id
|
||||
|
||||
# ── Token management ──────────────────────────────────────────
|
||||
|
||||
async def _refresh_token(self) -> None:
|
||||
"""Fetch or refresh the access_token."""
|
||||
if self._backend == "wecom":
|
||||
url = (
|
||||
f"https://qyapi.weixin.qq.com/cgi-bin/gettoken"
|
||||
f"?corpid={self.config.corp_id}"
|
||||
f"&corpsecret={self.config.secret}"
|
||||
)
|
||||
else:
|
||||
url = (
|
||||
f"https://api.weixin.qq.com/cgi-bin/token"
|
||||
f"?grant_type=client_credential"
|
||||
f"&appid={self.config.app_id}"
|
||||
f"&secret={self.config.app_secret}"
|
||||
)
|
||||
|
||||
try:
|
||||
resp = await self._http_client.get(url)
|
||||
data = resp.json()
|
||||
except Exception as e:
|
||||
if not self._running:
|
||||
raise ChannelError(f"Failed to get WeChat access token: {e}") from e
|
||||
raise RuntimeError(f"Failed to get WeChat access token: {e}") from e
|
||||
|
||||
if data.get("errcode", 0) != 0:
|
||||
err_msg = (
|
||||
f"WeChat auth error ({data.get('errcode')}): "
|
||||
f"{data.get('errmsg', 'unknown')}"
|
||||
)
|
||||
if not self._running:
|
||||
raise ChannelError(err_msg)
|
||||
raise RuntimeError(err_msg)
|
||||
|
||||
self._access_token = data["access_token"]
|
||||
expire = data.get("expires_in", 7200)
|
||||
# Refresh 5 minutes before expiry
|
||||
self._token_expires = time.monotonic() + expire - 300
|
||||
logger.debug(f"WeChat token refreshed, expires in {expire}s")
|
||||
|
||||
async def _ensure_token(self) -> str:
|
||||
"""Return a valid access token, refreshing if needed."""
|
||||
if not self._access_token or time.monotonic() >= self._token_expires:
|
||||
await self._refresh_token()
|
||||
return self._access_token
|
||||
|
||||
# ── Signature verification (GET callback) ─────────────────────
|
||||
|
||||
async def _handle_verify(self, request) -> web.Response:
|
||||
"""Handle GET /wechat/callback for URL verification.
|
||||
|
||||
WeChat/WeCom sends: msg_signature, timestamp, nonce, echostr
|
||||
We decrypt echostr (encrypted mode) or verify signature (plain mode)
|
||||
and return the plain echostr.
|
||||
"""
|
||||
from aiohttp import web
|
||||
|
||||
signature = request.query.get("msg_signature") or request.query.get(
|
||||
"signature", ""
|
||||
)
|
||||
timestamp = request.query.get("timestamp", "")
|
||||
nonce = request.query.get("nonce", "")
|
||||
echostr = request.query.get("echostr", "")
|
||||
|
||||
logger.info(f"Verify request received: timestamp={timestamp}")
|
||||
|
||||
if not echostr:
|
||||
return web.Response(status=400, text="missing echostr")
|
||||
|
||||
# Encrypted mode: WeCom sends msg_signature and encrypted echostr
|
||||
if self._crypto and request.query.get("msg_signature"):
|
||||
# Verify signature first
|
||||
sig_ok = self._crypto.verify_signature(signature, timestamp, nonce, echostr)
|
||||
if not sig_ok:
|
||||
logger.warning("WeChat verify: signature mismatch")
|
||||
# Try to decrypt regardless — the decrypted echostr must be returned
|
||||
try:
|
||||
plain_echostr, _ = self._crypto.decrypt(echostr)
|
||||
logger.info("WeChat verify: echostr decrypted successfully")
|
||||
return web.Response(text=plain_echostr)
|
||||
except Exception as e:
|
||||
logger.error(f"WeChat verify: echostr decrypt failed: {e}")
|
||||
return web.Response(status=500)
|
||||
else:
|
||||
# Plain mode verification
|
||||
token = self.config.token
|
||||
if token:
|
||||
parts = sorted([token, timestamp, nonce])
|
||||
expected = hashlib.sha1("".join(parts).encode()).hexdigest()
|
||||
if expected != signature:
|
||||
logger.warning("WeChat verify: signature mismatch (plain)")
|
||||
return web.Response(status=403)
|
||||
return web.Response(text=echostr)
|
||||
|
||||
# ── Inbound message handling (POST callback) ──────────────────
|
||||
|
||||
async def _handle_message(self, request) -> web.Response:
|
||||
"""Handle POST /wechat/callback for incoming messages."""
|
||||
from aiohttp import web
|
||||
|
||||
from .crypto import parse_xml
|
||||
|
||||
try:
|
||||
body = await request.text()
|
||||
except Exception:
|
||||
return web.Response(status=400)
|
||||
|
||||
logger.info(f"WeChat callback POST received, body length={len(body)}")
|
||||
xml_data = parse_xml(body)
|
||||
|
||||
# If encrypted, decrypt first
|
||||
encrypt = xml_data.get("Encrypt", "")
|
||||
if encrypt and self._crypto:
|
||||
signature = request.query.get("msg_signature", "")
|
||||
timestamp = request.query.get("timestamp", "")
|
||||
nonce = request.query.get("nonce", "")
|
||||
|
||||
if not self._crypto.verify_signature(signature, timestamp, nonce, encrypt):
|
||||
logger.warning("WeChat message signature mismatch")
|
||||
return web.Response(status=403)
|
||||
|
||||
try:
|
||||
decrypted_xml, _from_id = self._crypto.decrypt(encrypt)
|
||||
xml_data = parse_xml(decrypted_xml)
|
||||
except Exception as e:
|
||||
logger.error(f"WeChat decrypt failed: {e}")
|
||||
return web.Response(status=500)
|
||||
|
||||
# Process message asynchronously — WeCom requires a response within
|
||||
# 5 seconds, but media downloads can take much longer. Return
|
||||
# "success" immediately and handle the message in the background.
|
||||
_task = asyncio.create_task(self._safe_process_message(xml_data))
|
||||
self._background_tasks.add(_task)
|
||||
_task.add_done_callback(self._background_tasks.discard)
|
||||
|
||||
return web.Response(text="success")
|
||||
|
||||
async def _safe_process_message(self, xml_data: dict[str, str]) -> None:
|
||||
"""Wrapper that catches exceptions so fire-and-forget tasks don't leak."""
|
||||
try:
|
||||
await self._process_message(xml_data)
|
||||
except Exception:
|
||||
logger.exception("Error processing WeChat message")
|
||||
|
||||
async def _process_message(self, xml_data: dict[str, str]) -> None:
|
||||
"""Process a parsed XML message from WeChat/WeCom callback."""
|
||||
msg_type = xml_data.get("MsgType", "")
|
||||
from_user = xml_data.get("FromUserName", "")
|
||||
to_user = xml_data.get("ToUserName", "")
|
||||
content = xml_data.get("Content", "")
|
||||
msg_id = xml_data.get("MsgId", "")
|
||||
create_time = xml_data.get("CreateTime", "")
|
||||
|
||||
logger.info(
|
||||
f"WeChat message received: type={msg_type}, from={from_user}, id={msg_id}, keys={list(xml_data.keys())}"
|
||||
)
|
||||
|
||||
if not from_user:
|
||||
return
|
||||
|
||||
# Determine chat_id
|
||||
# For WeCom: FromUserName is the user's UserID
|
||||
# For MP: FromUserName is the user's OpenID
|
||||
chat_id = from_user
|
||||
|
||||
# Group chat detection
|
||||
is_group = False
|
||||
was_mentioned = True # Default: treat as mentioned (DMs)
|
||||
|
||||
# WeCom group detection: ChatId field indicates a group message
|
||||
if self._backend == "wecom":
|
||||
group_chat_id = xml_data.get("ChatId", "")
|
||||
if group_chat_id:
|
||||
is_group = True
|
||||
chat_id = group_chat_id
|
||||
# WeCom sets MsgType=event with Event=sys when bot is @mentioned,
|
||||
# but for text messages we check the XML AtUserList field
|
||||
at_user_list = xml_data.get("AtUserList", "")
|
||||
was_mentioned = bool(at_user_list)
|
||||
|
||||
# Handle different message types
|
||||
text = ""
|
||||
annotations: list[str] = []
|
||||
media_paths: list[str] = []
|
||||
|
||||
if msg_type == "text":
|
||||
text = content
|
||||
elif msg_type == "image":
|
||||
pic_url = xml_data.get("PicUrl", "")
|
||||
media_id = xml_data.get("MediaId", "")
|
||||
if pic_url:
|
||||
local, ann = await self._download_attachment(
|
||||
pic_url,
|
||||
f"wechat_{msg_id}.jpg",
|
||||
)
|
||||
if local:
|
||||
media_paths.append(local)
|
||||
if ann:
|
||||
annotations.append(ann)
|
||||
elif media_id:
|
||||
local, ann = await self._download_wechat_media(
|
||||
media_id,
|
||||
f"wechat_image_{msg_id}",
|
||||
)
|
||||
if local:
|
||||
media_paths.append(local)
|
||||
if ann:
|
||||
annotations.append(ann)
|
||||
else:
|
||||
annotations.append("[image: no download source]")
|
||||
elif msg_type == "voice":
|
||||
recognition = xml_data.get("Recognition", "")
|
||||
media_id = xml_data.get("MediaId", "")
|
||||
if media_id:
|
||||
local, ann = await self._download_wechat_media(
|
||||
media_id, f"wechat_voice_{msg_id}"
|
||||
)
|
||||
if local:
|
||||
media_paths.append(local)
|
||||
if ann:
|
||||
ann = ann.replace("[attachment:", "[voice:")
|
||||
annotations.append(ann)
|
||||
if recognition:
|
||||
text = f"[语音识别] {recognition}"
|
||||
elif not media_paths:
|
||||
annotations.append("[voice message]")
|
||||
elif msg_type in ("video", "shortvideo"):
|
||||
media_id = xml_data.get("MediaId", "")
|
||||
if media_id:
|
||||
local, ann = await self._download_wechat_media(
|
||||
media_id, f"wechat_{msg_type}_{msg_id}"
|
||||
)
|
||||
if local:
|
||||
media_paths.append(local)
|
||||
if ann:
|
||||
annotations.append(ann)
|
||||
if not media_paths:
|
||||
annotations.append(f"[{msg_type} message]")
|
||||
elif msg_type == "location":
|
||||
label = xml_data.get("Label", "")
|
||||
lat = xml_data.get("Location_X", "")
|
||||
lon = xml_data.get("Location_Y", "")
|
||||
text = f"[位置] {label} ({lat}, {lon})"
|
||||
elif msg_type == "file":
|
||||
media_id = xml_data.get("MediaId", "")
|
||||
file_name = xml_data.get("FileName", "") or xml_data.get(
|
||||
"Title", f"wechat_file_{msg_id}"
|
||||
)
|
||||
logger.info(
|
||||
f"WeChat file message: name={file_name}, media_id={media_id!r}, keys={list(xml_data.keys())}"
|
||||
)
|
||||
if media_id:
|
||||
local, ann = await self._download_wechat_media(
|
||||
media_id,
|
||||
f"wechat_file_{msg_id}_{file_name}",
|
||||
)
|
||||
logger.info(f"WeChat file download result: local={local}, ann={ann}")
|
||||
if local:
|
||||
media_paths.append(local)
|
||||
if ann:
|
||||
annotations.append(ann)
|
||||
if not media_paths:
|
||||
annotations.append(f"[file: {file_name}]")
|
||||
elif msg_type == "link":
|
||||
title = xml_data.get("Title", "")
|
||||
description = xml_data.get("Description", "")
|
||||
url = xml_data.get("Url", "")
|
||||
text = f"[链接] {title}\n{description}\n{url}"
|
||||
elif msg_type == "event":
|
||||
event_type = xml_data.get("Event", "")
|
||||
if event_type == "subscribe":
|
||||
text = "[用户关注]"
|
||||
elif event_type == "unsubscribe":
|
||||
logger.info(f"User {from_user} unsubscribed")
|
||||
return # Don't process
|
||||
elif event_type == "CLICK":
|
||||
event_key = xml_data.get("EventKey", "")
|
||||
text = f"[菜单点击] {event_key}"
|
||||
elif event_type in ("LOCATION", "VIEW"):
|
||||
# Periodic location reports and menu-link clicks — ignore
|
||||
return
|
||||
else:
|
||||
logger.debug(f"Ignoring WeChat event: {event_type}")
|
||||
return
|
||||
else:
|
||||
text = f"[{msg_type} message]"
|
||||
|
||||
if not text and not media_paths and not annotations:
|
||||
return
|
||||
|
||||
# Parse timestamp
|
||||
try:
|
||||
timestamp = (
|
||||
datetime.fromtimestamp(int(create_time))
|
||||
if create_time
|
||||
else datetime.now()
|
||||
)
|
||||
except (ValueError, TypeError, OSError):
|
||||
timestamp = datetime.now()
|
||||
|
||||
await self._enqueue_raw(
|
||||
RawIncoming(
|
||||
sender_id=from_user,
|
||||
chat_id=chat_id,
|
||||
text=text,
|
||||
media_files=media_paths,
|
||||
content_annotations=annotations,
|
||||
timestamp=timestamp,
|
||||
message_id=msg_id,
|
||||
is_group=is_group,
|
||||
was_mentioned=was_mentioned,
|
||||
metadata={
|
||||
"chat_id": chat_id,
|
||||
"to_user": to_user,
|
||||
"backend": self._backend,
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
# ── Send (template method overrides) ──────────────────────────
|
||||
|
||||
def _format_chunk(self, text: str) -> str:
|
||||
"""WeCom uses markdown formatter; MP uses plain text."""
|
||||
if self._backend == "wecom":
|
||||
return self._formatter.format(text) # markdown profile
|
||||
return _strip_markdown(text)
|
||||
|
||||
async def _send_chunk(
|
||||
self,
|
||||
chat_id,
|
||||
formatted_text,
|
||||
raw_text,
|
||||
reply_to,
|
||||
metadata,
|
||||
):
|
||||
token = await self._ensure_token()
|
||||
|
||||
if self._backend == "wecom":
|
||||
# Group chat: use appchat/send endpoint
|
||||
if chat_id.startswith("wr"):
|
||||
try:
|
||||
await self._wecom_send_group_markdown(token, chat_id, raw_text)
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
await self._wecom_send_group_text(token, chat_id, raw_text)
|
||||
else:
|
||||
# DM: Try markdown first, fall back to plain text
|
||||
try:
|
||||
await self._wecom_send_markdown(token, chat_id, raw_text)
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
await self._wecom_send_text(token, chat_id, raw_text)
|
||||
else:
|
||||
await self._mp_send_text(token, chat_id, raw_text)
|
||||
|
||||
# ── WeCom send ────────────────────────────────────────────────
|
||||
|
||||
async def _wecom_send_text(
|
||||
self,
|
||||
token: str,
|
||||
user_id: str,
|
||||
text: str,
|
||||
) -> None:
|
||||
"""Send a text message via WeCom API."""
|
||||
url = f"https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token={token}"
|
||||
body = {
|
||||
"touser": user_id,
|
||||
"msgtype": "text",
|
||||
"agentid": int(self.config.agent_id),
|
||||
"text": {"content": _strip_markdown(text)},
|
||||
}
|
||||
await self._post_api(url, body)
|
||||
|
||||
async def _wecom_send_markdown(
|
||||
self,
|
||||
token: str,
|
||||
user_id: str,
|
||||
text: str,
|
||||
) -> None:
|
||||
"""Send a markdown message via WeCom API.
|
||||
|
||||
Note: WeCom markdown only supports a subset of Markdown
|
||||
(no code blocks, no images). Falls back to text if the
|
||||
message is too complex.
|
||||
"""
|
||||
url = f"https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token={token}"
|
||||
body = {
|
||||
"touser": user_id,
|
||||
"msgtype": "markdown",
|
||||
"agentid": int(self.config.agent_id),
|
||||
"markdown": {"content": text},
|
||||
}
|
||||
await self._post_api(url, body)
|
||||
|
||||
# ── WeCom group send ────────────────────────────────────────────
|
||||
|
||||
async def _wecom_send_group_text(
|
||||
self,
|
||||
token: str,
|
||||
chatid: str,
|
||||
text: str,
|
||||
) -> None:
|
||||
"""Send a text message to a WeCom group chat."""
|
||||
url = f"https://qyapi.weixin.qq.com/cgi-bin/appchat/send?access_token={token}"
|
||||
body = {
|
||||
"chatid": chatid,
|
||||
"msgtype": "text",
|
||||
"text": {"content": _strip_markdown(text)},
|
||||
}
|
||||
await self._post_api(url, body)
|
||||
|
||||
async def _wecom_send_group_markdown(
|
||||
self,
|
||||
token: str,
|
||||
chatid: str,
|
||||
text: str,
|
||||
) -> None:
|
||||
"""Send a markdown message to a WeCom group chat."""
|
||||
url = f"https://qyapi.weixin.qq.com/cgi-bin/appchat/send?access_token={token}"
|
||||
body = {
|
||||
"chatid": chatid,
|
||||
"msgtype": "markdown",
|
||||
"markdown": {"content": text},
|
||||
}
|
||||
await self._post_api(url, body)
|
||||
|
||||
# ── MP send ───────────────────────────────────────────────────
|
||||
|
||||
async def _mp_send_text(
|
||||
self,
|
||||
token: str,
|
||||
openid: str,
|
||||
text: str,
|
||||
) -> None:
|
||||
"""Send a text message via WeChat MP customer service API."""
|
||||
url = (
|
||||
f"https://api.weixin.qq.com/cgi-bin/message/custom/send"
|
||||
f"?access_token={token}"
|
||||
)
|
||||
body = {
|
||||
"touser": openid,
|
||||
"msgtype": "text",
|
||||
"text": {"content": _strip_markdown(text)},
|
||||
}
|
||||
await self._post_api(url, body)
|
||||
|
||||
# ── Media send ────────────────────────────────────────────────
|
||||
|
||||
async def _send_media_impl(
|
||||
self,
|
||||
recipient: str,
|
||||
file_path: str,
|
||||
caption: str = "",
|
||||
metadata: dict | None = None,
|
||||
) -> bool:
|
||||
"""Send a media file via WeChat/WeCom."""
|
||||
token = await self._ensure_token()
|
||||
chat_id = self._resolve_media_chat_id(recipient, metadata)
|
||||
|
||||
# Upload media to get media_id
|
||||
media_id = await self._upload_media(token, file_path)
|
||||
if not media_id:
|
||||
return False
|
||||
|
||||
path = Path(file_path)
|
||||
ext = path.suffix.lower()
|
||||
is_image = ext in {".jpg", ".jpeg", ".png", ".gif", ".bmp"}
|
||||
|
||||
if self._backend == "wecom":
|
||||
msg_type = "image" if is_image else "file"
|
||||
# Group chat: use appchat/send endpoint
|
||||
if chat_id.startswith("wr"):
|
||||
url = (
|
||||
f"https://qyapi.weixin.qq.com/cgi-bin/appchat/send"
|
||||
f"?access_token={token}"
|
||||
)
|
||||
body = {
|
||||
"chatid": chat_id,
|
||||
"msgtype": msg_type,
|
||||
msg_type: {"media_id": media_id},
|
||||
}
|
||||
else:
|
||||
url = (
|
||||
f"https://qyapi.weixin.qq.com/cgi-bin/message/send"
|
||||
f"?access_token={token}"
|
||||
)
|
||||
body = {
|
||||
"touser": chat_id,
|
||||
"msgtype": msg_type,
|
||||
"agentid": int(self.config.agent_id),
|
||||
msg_type: {"media_id": media_id},
|
||||
}
|
||||
else:
|
||||
url = (
|
||||
f"https://api.weixin.qq.com/cgi-bin/message/custom/send"
|
||||
f"?access_token={token}"
|
||||
)
|
||||
msg_type = "image" if is_image else "file" # MP only supports image
|
||||
if not is_image:
|
||||
# MP doesn't support file via customer service API;
|
||||
# send caption as text instead
|
||||
if caption:
|
||||
await self._mp_send_text(
|
||||
token, chat_id, f"[文件] {path.name}\n{caption}"
|
||||
)
|
||||
return True
|
||||
body = {
|
||||
"touser": chat_id,
|
||||
"msgtype": "image",
|
||||
"image": {"media_id": media_id},
|
||||
}
|
||||
|
||||
await self._post_api(url, body)
|
||||
|
||||
# Send caption separately if provided
|
||||
if caption:
|
||||
if self._backend == "wecom":
|
||||
if chat_id.startswith("wr"):
|
||||
await self._wecom_send_group_text(token, chat_id, caption)
|
||||
else:
|
||||
await self._wecom_send_text(token, chat_id, caption)
|
||||
else:
|
||||
await self._mp_send_text(token, chat_id, caption)
|
||||
|
||||
return True
|
||||
|
||||
async def _upload_media(
|
||||
self,
|
||||
token: str,
|
||||
file_path: str,
|
||||
) -> str | None:
|
||||
"""Upload a media file and return the media_id."""
|
||||
path = Path(file_path)
|
||||
ext = path.suffix.lower()
|
||||
is_image = ext in {".jpg", ".jpeg", ".png", ".gif", ".bmp"}
|
||||
media_type = "image" if is_image else "file"
|
||||
|
||||
if self._backend == "wecom":
|
||||
url = (
|
||||
f"https://qyapi.weixin.qq.com/cgi-bin/media/upload"
|
||||
f"?access_token={token}&type={media_type}"
|
||||
)
|
||||
else:
|
||||
url = (
|
||||
f"https://api.weixin.qq.com/cgi-bin/media/upload"
|
||||
f"?access_token={token}&type={media_type}"
|
||||
)
|
||||
|
||||
try:
|
||||
with open(file_path, "rb") as f:
|
||||
resp = await self._http_client.post(
|
||||
url,
|
||||
files={"media": (path.name, f)},
|
||||
)
|
||||
data = resp.json()
|
||||
if data.get("errcode", 0) != 0 and "media_id" not in data:
|
||||
logger.error(f"WeChat media upload failed: {data.get('errmsg')}")
|
||||
return None
|
||||
return data.get("media_id")
|
||||
except Exception as e:
|
||||
logger.error(f"WeChat media upload error: {e}")
|
||||
return None
|
||||
|
||||
# ── Media download helper ────────────────────────────────────
|
||||
|
||||
async def _download_wechat_media(
|
||||
self,
|
||||
media_id: str,
|
||||
filename: str,
|
||||
) -> tuple[str | None, str | None]:
|
||||
"""Download media by media_id via WeChat/WeCom media API."""
|
||||
token = await self._ensure_token()
|
||||
if self._backend == "wecom":
|
||||
url = f"https://qyapi.weixin.qq.com/cgi-bin/media/get?access_token={token}&media_id={media_id}"
|
||||
else:
|
||||
url = f"https://api.weixin.qq.com/cgi-bin/media/get?access_token={token}&media_id={media_id}"
|
||||
return await self._download_attachment(url, filename)
|
||||
|
||||
# ── Shared API helper ─────────────────────────────────────────
|
||||
|
||||
async def _post_api(self, url: str, body: dict) -> dict:
|
||||
"""POST to WeChat/WeCom API, check errcode, return response."""
|
||||
try:
|
||||
resp = await self._http_client.post(url, json=body)
|
||||
data = resp.json()
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"WeChat API error: {e}") from e
|
||||
|
||||
errcode = data.get("errcode", 0)
|
||||
if errcode != 0:
|
||||
errmsg = data.get("errmsg", "unknown")
|
||||
# Token expired — refresh and retry once
|
||||
if errcode in (40014, 42001):
|
||||
logger.warning("WeChat token expired, refreshing...")
|
||||
await self._refresh_token()
|
||||
token = self._access_token
|
||||
# Replace token in URL
|
||||
if "access_token=" in url:
|
||||
url = re.sub(
|
||||
r"access_token=[^&]+",
|
||||
f"access_token={token}",
|
||||
url,
|
||||
)
|
||||
resp = await self._http_client.post(url, json=body)
|
||||
data = resp.json()
|
||||
if data.get("errcode", 0) != 0:
|
||||
raise RuntimeError(
|
||||
f"WeChat API error after retry: {data.get('errmsg')}"
|
||||
)
|
||||
return data
|
||||
else:
|
||||
raise RuntimeError(f"WeChat API error ({errcode}): {errmsg}")
|
||||
|
||||
return data
|
||||
|
||||
# ── Typing indicator (WeCom only) ────────────────────────────
|
||||
|
||||
async def _send_typing_action(self, chat_id: str) -> None:
|
||||
"""Send typing indicator via WeChat.
|
||||
|
||||
WeChat has no native typing API. For the WeCom backend we
|
||||
approximate the experience by posting a short-lived "…" message
|
||||
that is recalled once the real reply is sent (handled by
|
||||
``stop_typing``). WeChat MP has no recall API so we skip it.
|
||||
"""
|
||||
if self._backend != "wecom" or not self._http_client:
|
||||
return
|
||||
try:
|
||||
token = await self._ensure_token()
|
||||
if chat_id.startswith("wr"):
|
||||
url = (
|
||||
f"https://qyapi.weixin.qq.com/cgi-bin/appchat/send"
|
||||
f"?access_token={token}"
|
||||
)
|
||||
body = {
|
||||
"chatid": chat_id,
|
||||
"msgtype": "text",
|
||||
"text": {"content": "\u2026"},
|
||||
}
|
||||
else:
|
||||
url = (
|
||||
f"https://qyapi.weixin.qq.com/cgi-bin/message/send"
|
||||
f"?access_token={token}"
|
||||
)
|
||||
body = {
|
||||
"touser": chat_id,
|
||||
"msgtype": "text",
|
||||
"agentid": int(self.config.agent_id),
|
||||
"text": {"content": "\u2026"},
|
||||
}
|
||||
data = await self._post_api(url, body)
|
||||
msgid = data.get("msgid")
|
||||
if msgid:
|
||||
self._typing_message_ids.setdefault(chat_id, []).append(msgid)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def stop_typing(self, chat_id: str) -> None:
|
||||
"""Cancel typing loop and recall all status messages."""
|
||||
msgids = self._typing_message_ids.pop(chat_id, [])
|
||||
if msgids and self._http_client and self._backend == "wecom":
|
||||
try:
|
||||
token = await self._ensure_token()
|
||||
url = (
|
||||
f"https://qyapi.weixin.qq.com/cgi-bin/message/recall"
|
||||
f"?access_token={token}"
|
||||
)
|
||||
for msgid in msgids:
|
||||
try:
|
||||
await self._http_client.post(url, json={"msgid": msgid})
|
||||
except Exception:
|
||||
pass
|
||||
except Exception:
|
||||
pass
|
||||
await super().stop_typing(chat_id)
|
||||