Initial commit of EvoScientist framework

Self-evolving AI scientist framework built on LangGraph/LangChain with
CLI/TUI core, FastAPI gateway, and Next.js frontend.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
m4
2026-07-13 08:07:45 +08:00
commit 3c759b36cb
332 changed files with 99117 additions and 0 deletions
+27
View File
@@ -0,0 +1,27 @@
# EvoScientist CLI environment variables
# The preferred configuration flow is `evosci onboard`, which writes
# ~/.evoscientist/config/settings.yaml. Environment variables can override it.
# Optional application directories
# EVOSCIENTIST_HOME=~/.evoscientist
# EVOSCIENTIST_DATA_ROOT=~/.evoscientist/data
# Logging
EVOSCIENTIST_LOG_LEVEL=INFO
# EVOSCIENTIST_LOG_DIR=~/.evoscientist/data/logs
EVOSCIENTIST_LOG_RETENTION_DAYS=30
# Optional PostgreSQL checkpoint storage. Without this value the CLI uses an
# in-memory checkpointer for the current process.
# EVOSCIENTIST_SESSION_DB_URL=postgresql://user:password@localhost:5432/evoscientist
# Model provider examples. Provider-specific settings can also be configured
# through `evosci onboard`.
# OPENAI_API_KEY=sk-...
# OPENAI_BASE_URL=https://api.openai.com/v1
# ANTHROPIC_API_KEY=sk-ant-...
# GOOGLE_API_KEY=...
# TAVILY_API_KEY=tvly-...
# Optional default model
# DEFAULT_MODEL=openai/gpt-5.4
+38
View File
@@ -0,0 +1,38 @@
---
name: Bug report
about: Report a bug to help us improve EvoScientist
title: ''
labels: bug
assignees: ''
type: Bug
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Run `EvoSci ...`
2. Provide input '...'
3. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Error output / logs**
If applicable, paste the relevant terminal output or traceback.
```
<paste here>
```
**System information**
- OS: [e.g. Ubuntu 22.04, macOS 14.3, Windows 11]
- Python version: [e.g. 3.11.8]
- Terminal: [e.g. iTerm2, Alacritty, Windows Terminal]
- EvoScientist version: [e.g. 0.0.2 — run `EvoSci --version`]
- LLM provider: [e.g. Anthropic, OpenAI, Ollama]
**Additional context**
Add any other context about the problem here.
+8
View File
@@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: EvoSkills — niche features & custom workflows
url: https://github.com/EvoScientist/EvoSkills
about: Want to add a feature that serves a specific use case rather than all users? Contribute it as an EvoSkill instead.
- name: Community Discord
url: https://discord.gg/AZ9ZMXkunY
about: Ask questions, get help, and chat with the community.
+18
View File
@@ -0,0 +1,18 @@
---
name: Documentation
about: Report missing, incorrect, or unclear documentation
title: ''
labels: documentation
assignees: ''
type: Documentation
---
**What documentation is affected?**
Describe which part of the project needs better docs.
**What is wrong or missing?**
Describe the issue — is something inaccurate, outdated, hard to follow, or missing entirely?
**Suggested improvement**
If you have a specific fix or addition in mind, describe it here.
+24
View File
@@ -0,0 +1,24 @@
---
name: Feature request
about: Suggest an idea for EvoScientist core functionality
title: ''
labels: enhancement
assignees: ''
type: Feature
---
> **Note:** We only accept feature requests for core functionality that benefits the majority of users.
> If your idea is a niche workflow or personal utility, consider contributing it to [EvoSkills](https://github.com/EvoScientist/EvoSkills) instead.
**Is your feature request related to a problem? Please describe.**
A clear and concise description of the problem. Ex. "I'm always frustrated when [...]"
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
+21
View File
@@ -0,0 +1,21 @@
## Description
<!-- What does this PR do? Link the related issue (e.g. "Closes #123"). -->
## Type of change
<!-- Check the one that applies. -->
- [ ] Bug fix
- [ ] New feature — link issue: #<!-- issue number -->
- [ ] Documentation / examples
- [ ] Test improvement
- [ ] Refactor (no behavior change)
## Checklist
- [ ] I have read the [Contributing Guidelines](../CONTRIBUTING.md)
- [ ] This targets **core functionality** used by the majority of users (niche features belong in [EvoSkills](https://github.com/EvoScientist/EvoSkills))
- [ ] I have added/updated tests where applicable
- [ ] `uv run ruff check .` passes
- [ ] `uv run pytest` passes
Binary file not shown.

After

Width:  |  Height:  |  Size: 849 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 316 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 540 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 284 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 582 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 434 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 642 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 739 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.9 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 370 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 492 KiB

+9
View File
@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" width="188" height="34">
<rect x="0.5" y="0.5" width="187" height="33" rx="8" fill="#ffffff" stroke="#e5e7eb"/>
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="500" fill="#6b7280">Built with</text>
<rect x="84" y="5" width="96" height="24" rx="6" fill="#7c3aed"/>
<text x="132" y="22" text-anchor="middle"
font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="700" fill="#ffffff">DeepAgents</text>
</svg>

After

Width:  |  Height:  |  Size: 566 B

+9
View File
@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" width="188" height="34">
<rect x="0.5" y="0.5" width="187" height="33" rx="8" fill="#1e293b" stroke="#334155"/>
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="500" fill="#94a3b8">Built with</text>
<rect x="84" y="5" width="96" height="24" rx="6" fill="#8b5cf6"/>
<text x="132" y="22" text-anchor="middle"
font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="700" fill="#ffffff">DeepAgents</text>
</svg>

After

Width:  |  Height:  |  Size: 566 B

+9
View File
@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" width="168" height="34">
<rect x="0.5" y="0.5" width="167" height="33" rx="8" fill="#ffffff" stroke="#e5e7eb"/>
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="500" fill="#6b7280">License</text>
<rect x="71" y="5" width="89" height="24" rx="6" fill="#059669"/>
<text x="115.5" y="22" text-anchor="middle"
font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="700" fill="#ffffff">Apache 2.0</text>
</svg>

After

Width:  |  Height:  |  Size: 565 B

+9
View File
@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" width="168" height="34">
<rect x="0.5" y="0.5" width="167" height="33" rx="8" fill="#1e293b" stroke="#334155"/>
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="500" fill="#94a3b8">License</text>
<rect x="71" y="5" width="89" height="24" rx="6" fill="#10b981"/>
<text x="115.5" y="22" text-anchor="middle"
font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="700" fill="#ffffff">Apache 2.0</text>
</svg>

After

Width:  |  Height:  |  Size: 565 B

+9
View File
@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" width="124" height="34">
<rect x="0.5" y="0.5" width="123" height="33" rx="8" fill="#ffffff" stroke="#e5e7eb"/>
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="500" fill="#6b7280">PyPI</text>
<rect x="54" y="5" width="62" height="24" rx="6" fill="#1565c0"/>
<text x="85" y="22" text-anchor="middle"
font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="700" fill="#ffffff">v0.0.7</text>
</svg>

After

Width:  |  Height:  |  Size: 555 B

+9
View File
@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" width="124" height="34">
<rect x="0.5" y="0.5" width="123" height="33" rx="8" fill="#1e293b" stroke="#334155"/>
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="500" fill="#94a3b8">PyPI</text>
<rect x="54" y="5" width="62" height="24" rx="6" fill="#2563eb"/>
<text x="85" y="22" text-anchor="middle"
font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="700" fill="#ffffff">v0.0.7</text>
</svg>

After

Width:  |  Height:  |  Size: 555 B

+9
View File
@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" width="136" height="34">
<rect x="0.5" y="0.5" width="135" height="33" rx="8" fill="#ffffff" stroke="#e5e7eb"/>
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="500" fill="#6b7280">Website</text>
<rect x="74" y="5" width="54" height="24" rx="6" fill="#ea580c"/>
<text x="101" y="22" text-anchor="middle"
font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="700" fill="#ffffff">Page</text>
</svg>

After

Width:  |  Height:  |  Size: 557 B

+9
View File
@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" width="136" height="34">
<rect x="0.5" y="0.5" width="135" height="33" rx="8" fill="#1e293b" stroke="#334155"/>
<text x="15" y="22" font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="500" fill="#94a3b8">Website</text>
<rect x="74" y="5" width="54" height="24" rx="6" fill="#f97316"/>
<text x="101" y="22" text-anchor="middle"
font-family="Inter, -apple-system, system-ui, sans-serif"
font-size="13" font-weight="700" fill="#ffffff">Page</text>
</svg>

After

Width:  |  Height:  |  Size: 557 B

+7
View File
@@ -0,0 +1,7 @@
# 📱 EvoScientist 微信二维码
### 👥 微信交流群:
<!-- ![微信群二维码](./wechat_group.jpeg) -->
<!-- 3-23 -->
<img src="./wechat_group.jpeg" alt="微信群二维码" width="300"/>
Binary file not shown.

After

Width:  |  Height:  |  Size: 654 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 94 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 94 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 428 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

+24
View File
@@ -0,0 +1,24 @@
name: Build
on:
push:
branches: ["main"]
pull_request:
jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v5
- uses: astral-sh/setup-uv@v6
with:
python-version: "3.11"
cache-dependency-glob: "**/pyproject.toml"
- name: Build package
run: uv build
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: dist
path: dist/
+23
View File
@@ -0,0 +1,23 @@
name: Lint
on:
push:
branches: ["main"]
pull_request:
jobs:
ruff:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v5
- uses: astral-sh/setup-uv@v6
with:
python-version: "3.11"
cache-dependency-glob: "**/pyproject.toml"
- name: Install dependencies
run: uv sync --dev
- name: Run ruff linter
run: uv run ruff check --output-format=github .
- name: Run ruff format
run: uv run ruff format --check .
+24
View File
@@ -0,0 +1,24 @@
name: Test
on:
push:
branches: ["main"]
pull_request:
jobs:
pytest:
runs-on: ubuntu-latest
timeout-minutes: 15
strategy:
matrix:
python-version: ["3.11", "3.12"]
steps:
- uses: actions/checkout@v5
- uses: astral-sh/setup-uv@v6
with:
python-version: ${{ matrix.python-version }}
cache-dependency-glob: "**/pyproject.toml"
- name: Install dependencies
run: uv sync --dev
- name: Run pytest
run: uv run pytest -v --timeout=30
+67
View File
@@ -0,0 +1,67 @@
# macOS
.DS_Store
# Python
__pycache__/
*.py[cod]
*.egg-info/
dist/
build/
*.egg
*.pytest_cache/
.coverage
.ipynb_checkpoints/
# Environment
.env
.env.*
.env_*
!.env.example
.venv/
venv/
bridge/node_modules/
bridge/package-lock.json
# IDE / Tools
.vscode/
.idea/
.agents/
.claude/
.cursor/
.codex/
.cursorrules
.ruff_cache/
# Project specific
.langgraph_api/
workspace/
skills/
memory/
media/
conversation_history/
.deno_cache/
*.ipynb
*CLAUDE.md
*AGENTS.md
*meals/
botpy.log
large_tool_results/
# Docker runtime data
docker/data/
# Project-level config data
.data/
# Sensitive / credentials (never commit)
postgresql:*
_s3_backup/
# Local scratch / tooling data
.superpowers/
.test-home/
tmp/
# Root-level debug scratch scripts (proper tests live in tests/)
/test_*.py
/research_lookup_temp.py
+10
View File
@@ -0,0 +1,10 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.9.9
hooks:
# Run the linter.
- id: ruff
args: [ --fix ]
# Run the formatter.
- id: ruff-format
+111
View File
@@ -0,0 +1,111 @@
# Contributing to EvoScientist
We appreciate your interest and the time you spend helping improve EvoScientist. Please read the following guidelines before contributing.
## How you can contribute
- **Report bugs and request features:** open an issue using the provided templates. Make sure to use the correct template and labels.
- **Propose design changes:** use issues or discussion threads to outline the problem, alternatives, and trade-offs before implementing.
- **Contribute code or docs:** submit PRs that address an open issue. They must have a clear rationale and tests where applicable.
## What we are looking for in PRs
We aim to keep EvoScientist focused on core functionality that benefits the majority of users. PRs should only include:
- Bug fixes / improvements to existing features
- New features that were proposed in an issue and agreed upon with maintainers
- Documentation updates and examples
- Meaningful additions to the test suite
If you want to add a niche or specialized workflow, consider contributing to the [EvoSkills repository](https://github.com/EvoScientist/EvoSkills) instead.
## Development setup
1. **Fork and clone** the repository:
```bash
git clone https://github.com/<your-username>/EvoScientist.git
cd EvoScientist
```
2. **Install dependencies** (requires [uv](https://docs.astral.sh/uv/)):
```bash
uv sync --dev
```
3. **Run the test suite** (no API keys needed):
```bash
uv run pytest
```
4. **Run the linter:**
```bash
uv run ruff check .
```
## Submitting a pull request
1. Create a branch from `main` with a descriptive name (e.g. `fix/session-crash`, `feat/export-csv`).
2. Make your changes, keeping commits focused and well-described.
3. Ensure `uv run ruff check .` and `uv run pytest` pass locally — these also run in CI.
4. Open a PR against `main` and fill in the PR template.
5. A maintainer will review your PR. Please be responsive to feedback.
## Code style
- We use [Ruff](https://docs.astral.sh/ruff/) for linting. Run `uv run ruff check .` before pushing.
- Follow the existing code patterns and conventions in the area you're modifying.
- Keep changes minimal and focused on the task at hand.
---
## Project overview
EvoScientist is a multi-agent AI system for automated scientific experimentation and discovery. It orchestrates specialized sub-agents that plan experiments, search literature, write code, debug, analyze data, and draft reports.
| Fact | Value |
|------|-------|
| Language | Python 3.11+ |
| License | Apache 2.0 |
| Framework | [DeepAgents](https://github.com/langchain-ai/deepagents) + [LangChain](https://python.langchain.com/) + [LangGraph](https://langchain-ai.github.io/langgraph/) |
| Default model | `claude-sonnet-4-6` (Anthropic) |
| Tests | ~890 across 36 files, no API keys needed |
| Config file | `.data/.config/settings.yaml` (project root) |
### Sub-Agents (defined in `EvoScientist/subagent.yaml`)
| Agent | Purpose |
|-------|---------|
| `planner-agent` | Creates and updates experimental plans (no web search, no implementation) |
| `research-agent` | Web research for methods, baselines, and datasets (Tavily search) |
| `code-agent` | Implements experiment code and runnable scripts |
| `debug-agent` | Reproduces failures, identifies root causes, applies minimal fixes |
| `data-analysis-agent` | Computes metrics, creates plots, summarizes insights |
| `writing-agent` | Drafts paper-ready Markdown experiment reports |
### Data flow
```txt
User Input (CLI / TUI / 10 Channel Integrations)
|
CLI (cli/) / TUI (cli/tui_*) / Channel Server (channels/)
|
Main Agent (EvoScientist.py) -- create_deep_agent()
+-- System Prompt (prompts.py)
+-- Chat Model (llm/ -- multi-provider)
+-- Middleware: Memory (middleware/memory.py)
+-- Backend: CompositeBackend (backends.py)
| / --> CustomSandboxBackend (workspace read/write + execute)
| /skills/ --> MergedReadOnlyBackend (user > built-in)
| /memory/ --> FilesystemBackend (persistent cross-session)
+-- MCP Tools (mcp/ -- optional, cached by config signature)
|
task tool --> Delegates to Sub-Agents
|
Stream Events --> Emitter --> Tracker --> State --> Rich Display / TUI
```
---
## Need help?
Reach us on [Discord](https://discord.gg/AZ9ZMXkunY) or WeChat (linked in README).
+766
View File
@@ -0,0 +1,766 @@
"""EvoScientist Agent graph construction.
This module defines the agent graph and its factory functions. All heavy
initialization (deepagents, backends, LLM, middleware) is deferred to first
use so that importing this module is fast and non-agent CLI commands
(``EvoSci config list``, ``EvoSci onboard``) never pay the cost.
Usage:
from EvoScientist import EvoScientist_agent
# Notebook / programmatic usage
for state in EvoScientist_agent.stream(
{"messages": [HumanMessage(content="your question")]},
config={"configurable": {"thread_id": "1"}},
stream_mode="values",
):
...
"""
import json
import logging
import os
from datetime import datetime
from pathlib import Path
from typing import Any
from langchain.agents.middleware import AgentMiddleware
from . import paths as _paths_mod
from .config import apply_config_to_env, get_effective_config
from .paths import set_active_workspace, set_workspace_root
from .path_semantics import path_semantics_prompt
from .prompts import (
RESEARCHER_INSTRUCTIONS,
USER_FACING_LANGUAGE_POLICY,
get_system_prompt,
)
# Suppress noisy warnings from deepagents skill loader (non-string frontmatter fields, etc.)
logging.getLogger("deepagents.middleware.skills").setLevel(logging.ERROR)
logger = logging.getLogger(__name__)
# =============================================================================
# Constants
# =============================================================================
SUBAGENTS_CONFIG = Path(__file__).parent / "subagent.yaml"
SKILLS_DIR = str(Path(__file__).parent / "skills")
# =============================================================================
# Lazy state — initialized on first use, not at import time
# =============================================================================
_config = None
_chat_model = None
# Cache MCP tools by the effective config signature to avoid reconnecting
# to MCP servers on every `/new` when config is unchanged.
_MCP_TOOLS_CACHE_KEY: str | None = None
_MCP_TOOLS_CACHE_VALUE: dict[str, list] | None = None
# Default agent (no checkpointer) — used by langgraph dev / LangSmith / notebooks.
# Lazily constructed on first access so MCP tools are included without
# spawning subprocesses at import time.
_EvoScientist_agent = None
# =============================================================================
# Lazy initialization helpers
# =============================================================================
def reset_runtime_caches() -> None:
"""Clear cached default runtime objects after configuration changes."""
import sys
global _config, _chat_model, _EvoScientist_agent
_config = None
_chat_model = None
_EvoScientist_agent = None
package = sys.modules.get("EvoScientist")
if package is not None and "EvoScientist_agent" in vars(package):
delattr(package, "EvoScientist_agent")
def _ensure_config(config=None):
"""Return cached config. If *config* is passed, cache and use it."""
global _config
if config is not None:
_config = config
apply_config_to_env(_config)
if _config is None:
_config = get_effective_config()
apply_config_to_env(_config)
return _config
def _ensure_chat_model():
"""Return cached chat model, creating it on first call."""
global _chat_model
if _chat_model is None:
from .llm import get_chat_model
cfg = _ensure_config()
_chat_model = get_chat_model(model=cfg.model, provider=cfg.provider)
return _chat_model
# =============================================================================
# MCP caching
# =============================================================================
def _load_mcp_config_once() -> tuple[str, dict]:
"""Load MCP config and return ``(signature, config)``.
The signature includes the active workspace directory so that MCP
subprocesses are spawned with the correct ``cwd`` and environment
for each isolated workspace (e.g. per web thread).
"""
from .mcp.client import load_mcp_config
from .paths import _active_workspace
cfg = load_mcp_config()
if not cfg:
logger.info("MCP startup: no MCP servers configured")
return "", {}
logger.info(
"MCP startup: loaded %d server config(s): %s",
len(cfg),
", ".join(cfg.keys()),
)
try:
sig = json.dumps({"mcp": cfg, "workspace": str(_active_workspace)}, sort_keys=True, ensure_ascii=True)
except TypeError:
sig = repr(cfg) + str(_active_workspace)
return sig, cfg
def _load_mcp_tools_cached() -> dict[str, list]:
"""Load MCP tools with config + workspace-aware caching.
MCP subprocesses are spawned with ``cwd`` and ``EVOSCIENTIST_WORKSPACE_DIR``
set to the active workspace. Including the workspace in the cache key
ensures that each isolated workspace (e.g. per web thread) gets its own
MCP subprocess with the correct working directory.
"""
global _MCP_TOOLS_CACHE_KEY, _MCP_TOOLS_CACHE_VALUE
from .mcp import load_mcp_tools
cfg_key, cfg = _load_mcp_config_once()
if not cfg_key:
_MCP_TOOLS_CACHE_KEY = ""
_MCP_TOOLS_CACHE_VALUE = {}
return {}
if _MCP_TOOLS_CACHE_KEY == cfg_key and _MCP_TOOLS_CACHE_VALUE is not None:
logger.info(
"MCP startup: using cached MCP tools for %d agent route(s)",
len(_MCP_TOOLS_CACHE_VALUE),
)
return {k: list(v) for k, v in _MCP_TOOLS_CACHE_VALUE.items()}
loaded = load_mcp_tools(config=cfg)
if not loaded:
logger.warning("MCP startup: no tools loaded from configured MCP servers")
_MCP_TOOLS_CACHE_KEY = cfg_key
_MCP_TOOLS_CACHE_VALUE = {}
return {}
_MCP_TOOLS_CACHE_KEY = cfg_key
_MCP_TOOLS_CACHE_VALUE = {k: list(v) for k, v in loaded.items()}
return {k: list(v) for k, v in loaded.items()}
# =============================================================================
# Agent construction helpers
# =============================================================================
def _inject_subagent_middleware(subs: list[dict]) -> None:
"""Ensure every subagent gets error handling and context management middleware.
Without this, subagent tool errors are caught by LangGraph's default
ToolNode handler which produces terse messages without tracebacks or
retry guidance — reducing the subagent's ability to self-recover.
"""
from .middleware import (
ContextOverflowMapperMiddleware,
ToolErrorHandlerMiddleware,
create_context_editing_middleware,
)
for sa in subs:
sa.setdefault("middleware", []).extend(
[
# Uses main agent's model for trigger — subagents currently
# share the same model, so context window matches.
create_context_editing_middleware(),
ToolErrorHandlerMiddleware(),
ContextOverflowMapperMiddleware(),
]
)
def _inject_subagent_language_policy(subs: list[dict]) -> None:
"""Append the shared user-facing language policy to every subagent."""
for sa in subs:
prompt = str(sa.get("system_prompt") or "")
if "## User-Facing Language Policy" not in prompt:
sa["system_prompt"] = prompt.rstrip() + "\n\n" + USER_FACING_LANGUAGE_POLICY
def _inject_subagent_path_policy(subs: list[dict]) -> None:
"""Append the shared path semantics policy to every subagent."""
policy = path_semantics_prompt()
for sa in subs:
prompt = str(sa.get("system_prompt") or "")
if "## Path Semantics" not in prompt:
sa["system_prompt"] = prompt.rstrip() + "\n\n" + policy
def _build_prompt_refs() -> dict:
"""Build prompt references with the current date (not frozen at import)."""
return {
"RESEARCHER_INSTRUCTIONS": RESEARCHER_INSTRUCTIONS.format(
date=datetime.now().strftime("%Y-%m-%d"),
),
}
def _build_base_kwargs(base_backend, base_middleware, *, chat_model=None):
"""Build agent kwargs *without* MCP (fast, no subprocess spawning)."""
from .tools import (
edit_image,
generate_image,
refresh_image_tool_descriptions,
skill_manager,
think_tool,
web_crawl,
web_extract,
web_search,
)
from .utils import load_subagents
tool_registry = {"think_tool": think_tool}
# Register unified search tools when any search backend key is available
_has_search = any(os.environ.get(k) for k in (
"ZHIPU_API_KEY", "TAVILY_API_KEY", "FIRECRAWL_API_KEY", "FIRECRAWL_API_URL", "EXA_API_KEY",
))
if _has_search:
tool_registry["web_search"] = web_search
tool_registry["web_extract"] = web_extract
tool_registry["web_crawl"] = web_crawl
refresh_image_tool_descriptions()
base_tools = [think_tool, skill_manager, generate_image, edit_image]
subs = load_subagents(
SUBAGENTS_CONFIG,
tool_registry=tool_registry,
prompt_refs=_build_prompt_refs(),
)
_inject_subagent_path_policy(subs)
_inject_subagent_language_policy(subs)
_inject_subagent_middleware(subs)
return {
"name": "EvoScientist",
"model": chat_model if chat_model is not None else _ensure_chat_model(),
"tools": list(base_tools),
"backend": base_backend,
"subagents": subs,
"middleware": base_middleware,
"system_prompt": get_system_prompt(has_parse_documents=False),
"skills": ["/skills/"],
}
def _apply_conversation_profile(kwargs: dict, conversation_mode: str | None) -> dict:
"""Apply per-request capability limits without mutating shared inputs."""
from .conversation_modes import get_conversation_profile
profile = get_conversation_profile(conversation_mode)
if profile.enable_subagents:
return kwargs
adjusted = dict(kwargs)
adjusted["subagents"] = []
return adjusted
def _resolve_reasoning_effort(effort: str | None, cfg) -> str:
"""Resolve reasoning effort: explicit value wins, else fall back to config.
Decoupled from conversation mode — the profile no longer carries a
reasoning_effort default, so the fallback is the top-level config field.
"""
if effort is not None:
return effort
return getattr(cfg, "reasoning_effort", "medium")
def _create_tool_selector_for_profile(factory, threshold: int):
"""Call tool-selector factory with threshold, tolerating legacy test doubles."""
try:
return factory(threshold=threshold)
except TypeError as exc:
if "threshold" not in str(exc) and "unexpected keyword" not in str(exc):
raise
return factory()
def load_mcp_and_build_kwargs(base_backend, base_middleware, *, chat_model=None):
"""Load MCP tools (cached by config) and build agent kwargs.
Re-connects to MCP servers only when the effective MCP config changes.
Falls back to base kwargs if no MCP configured.
"""
from .tools import (
edit_image,
generate_image,
refresh_image_tool_descriptions,
skill_manager,
think_tool,
web_crawl,
web_extract,
web_search,
)
from .utils import load_subagents
mcp_by_agent = _load_mcp_tools_cached()
if not mcp_by_agent:
logger.info("MCP startup: agent will start without MCP tools")
return _build_base_kwargs(base_backend, base_middleware, chat_model=chat_model)
tool_registry = {"think_tool": think_tool}
# Register unified search tools when any search backend key is available
_has_search = any(os.environ.get(k) for k in (
"ZHIPU_API_KEY", "TAVILY_API_KEY", "FIRECRAWL_API_KEY", "FIRECRAWL_API_URL", "EXA_API_KEY",
))
if _has_search:
tool_registry["web_search"] = web_search
tool_registry["web_extract"] = web_extract
tool_registry["web_crawl"] = web_crawl
refresh_image_tool_descriptions()
base_tools = [think_tool, skill_manager, generate_image, edit_image]
# Fresh tool registry — start from base tools + MCP tools
registry = dict(tool_registry)
for tools in mcp_by_agent.values():
for t in tools:
registry[t.name] = t
mcp_main = mcp_by_agent.pop("main", [])
has_parse_documents = any(t.name == "parse_documents" for t in mcp_main)
logger.info(
"MCP startup: injecting %d MCP tool(s) into main agent: %s",
len(mcp_main),
", ".join(t.name for t in mcp_main) or "(none)",
)
subs = load_subagents(
SUBAGENTS_CONFIG,
tool_registry=registry,
prompt_refs=_build_prompt_refs(),
)
_inject_subagent_language_policy(subs)
_inject_subagent_path_policy(subs)
_inject_subagent_middleware(subs)
# Inject MCP tools into subagents by name
for sa in subs:
if sa_tools := mcp_by_agent.get(sa["name"], []):
sa.setdefault("tools", []).extend(sa_tools)
logger.info(
"MCP startup: injecting %d MCP tool(s) into subagent %s: %s",
len(sa_tools),
sa["name"],
", ".join(t.name for t in sa_tools),
)
return {
"name": "EvoScientist",
"model": chat_model if chat_model is not None else _ensure_chat_model(),
"tools": base_tools + mcp_main,
"backend": base_backend,
"subagents": subs,
"middleware": base_middleware,
"system_prompt": get_system_prompt(has_parse_documents=has_parse_documents),
"skills": ["/skills/"],
}
# =============================================================================
# Default agent (langgraph dev / notebooks)
# =============================================================================
def _get_default_backend():
"""Build the default composite backend from current paths."""
from deepagents.backends import CompositeBackend, FilesystemBackend
from .backends import CustomSandboxBackend, MergedReadOnlyBackend
workspace_dir = str(_paths_mod.WORKSPACE_ROOT)
set_active_workspace(workspace_dir)
memory_dir = str(_paths_mod.MEMORY_DIR)
user_skills_dir = str(_paths_mod.USER_SKILLS_DIR)
global_skills_dir = str(_paths_mod.GLOBAL_SKILLS_DIR)
ws_backend = CustomSandboxBackend(
root_dir=workspace_dir,
virtual_mode=True,
timeout=300,
)
sk_backend = MergedReadOnlyBackend(
primary_dir=user_skills_dir,
global_dir=global_skills_dir,
secondary_dir=SKILLS_DIR,
)
mem_backend = FilesystemBackend(
root_dir=memory_dir,
virtual_mode=True,
)
return CompositeBackend(
default=ws_backend,
routes={
"/skills/": sk_backend,
"/memory/": mem_backend,
},
)
def _get_default_middleware():
"""Build the default middleware list."""
from .middleware import (
ContextOverflowMapperMiddleware,
ToolErrorHandlerMiddleware,
create_context_editing_middleware,
create_memory_middleware,
create_tool_selector_middleware,
)
cfg = _ensure_config()
model = _ensure_chat_model()
memory_dir = str(_paths_mod.MEMORY_DIR)
mw: list[AgentMiddleware] = [
create_context_editing_middleware(model),
ContextOverflowMapperMiddleware(),
ToolErrorHandlerMiddleware(),
*create_tool_selector_middleware(),
create_memory_middleware(memory_dir, extraction_model=model),
]
if cfg.enable_ask_user and not cfg.auto_mode:
from .middleware.ask_user import AskUserMiddleware
mw.insert(0, AskUserMiddleware())
return mw
def _get_default_agent():
"""Build the default agent (with MCP, no checkpointer) on first access."""
global _EvoScientist_agent
if _EvoScientist_agent is None:
from deepagents import create_deep_agent
be = _get_default_backend()
mw = _get_default_middleware()
kwargs = load_mcp_and_build_kwargs(be, mw)
_EvoScientist_agent = create_deep_agent(**kwargs).with_config(
{"recursion_limit": 1000}
)
return _EvoScientist_agent
def __getattr__(name: str):
if name == "EvoScientist_agent":
return _get_default_agent()
# Backward compat for module-level names
if name == "chat_model":
return _ensure_chat_model()
if name == "SYSTEM_PROMPT":
return get_system_prompt(has_parse_documents=False)
if name == "backend":
return _get_default_backend()
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
# =============================================================================
# CLI agent factory
# =============================================================================
def create_cli_agent(
workspace_dir: str | None = None,
checkpointer=None,
config=None,
memory_dir: str | None = None,
source: str = "cli",
user_id: str | None = None,
model: str | None = None,
provider: str | None = None,
reasoning_effort: str | None = None,
# ── Compute (Phase 2B v1.6) ─────────────────────────────────────
compute_client=None, # ComputeClient for remote execution
storage_service=None, # StorageService for object storage
app_state=None, # Gateway FastAPI app.state
plan: str = "starter", # User plan for quota checks
thread_id: str = "", # Thread ID for compute billing
compute_quota_ok: bool = False,
remaining_compute_minutes: int = 0,
input_objects: list | None = None,
artifact_prefix: str = "",
default_output_paths: list[str] | None = None,
max_runtime_seconds: int | None = None,
budget_snapshot: dict | None = None,
storage_backend: str = "nfs",
execution_mode: str = "standard",
resource_class: str = "small",
backend_policy: str = "auto",
compute_fallback_mode: str = "local_sandbox",
conversation_mode: str | None = None,
_backend_ref: dict | None = None,
):
"""Create agent with checkpointer for CLI multi-turn support.
A fresh backend is constructed on every call using the current
``paths.WORKSPACE_ROOT`` (or the explicit *workspace_dir*), so
runtime ``set_workspace_root()`` changes are always respected.
Args:
workspace_dir: Per-session workspace directory. If ``None``,
defaults to the current ``paths.WORKSPACE_ROOT``.
checkpointer: Optional LangGraph checkpointer. If ``None``,
falls back to ``InMemorySaver`` (non-persistent).
config: Optional pre-loaded ``EvoScientistConfig``. If ``None``,
loads from file/env/defaults. Passing this avoids double
loading when the CLI has already loaded config.
memory_dir: Optional memory directory override. If ``None``,
uses the default global memory path. Gateway passes per-user path.
source: Thread source identifier (\"cli\" or \"web\"). Default \"cli\".
user_id: Optional user identifier for multi-tenant isolation.
model: Optional model override (short name or full ID). Gateway
passes the user-selected model from the frontend.
provider: Optional provider override. Inferred from *model* if
not specified.
reasoning_effort: Optional reasoning effort level for OpenAI models.
One of "low", "medium", "high". Passed to the LLM as
``reasoning={"effort": value, "summary": "auto"}``.
"""
import os as _os
from deepagents import create_deep_agent
from deepagents.backends import CompositeBackend, FilesystemBackend
from . import paths as _paths
from .backends import (
CustomSandboxBackend,
MergedReadOnlyBackend,
MultiRootSandboxBackend,
NonComputingFallbackBackend,
)
from .middleware import (
ContextOverflowMapperMiddleware,
ToolErrorHandlerMiddleware,
create_context_editing_middleware,
create_memory_middleware,
create_tool_selector_middleware,
ToolResultLimiterMiddleware,
)
cfg = _ensure_config(config)
from .conversation_modes import (
get_conversation_profile,
reset_current_conversation_mode,
set_current_conversation_mode,
)
mode_profile = get_conversation_profile(conversation_mode or getattr(cfg, "conversation_mode", "medium"))
if checkpointer is None:
from langgraph.checkpoint.memory import InMemorySaver
checkpointer = InMemorySaver()
# When no explicit workspace_dir is provided, apply config.default_workdir
# as a fallback. This covers direct callers (notebooks, iMessage server)
# that never call set_workspace_root() themselves. CLI callers always
# pass workspace_dir explicitly, so their --workdir is never overwritten.
if workspace_dir is None:
if cfg.default_workdir:
set_workspace_root(
_os.path.abspath(_os.path.expanduser(cfg.default_workdir))
)
workspace_dir = str(_paths.WORKSPACE_ROOT)
# Read paths dynamically so runtime set_workspace_root() changes are picked up
_mem_dir = memory_dir or str(_paths.MEMORY_DIR)
_usr_skills_dir = str(_paths.USER_SKILLS_DIR)
_global_skills_dir = str(_paths.GLOBAL_SKILLS_DIR)
# Always construct fresh backends from current paths (avoids stale
# module-level backend when workspace root changed at runtime).
set_active_workspace(workspace_dir)
if source == "web" and user_id:
# Web user: use remote compute only when the v1.6 preflight approved it.
if (
compute_client is not None
and getattr(compute_client, "ready", False)
and compute_quota_ok
and storage_backend in ("nfs", "file_gateway")
):
from .container_sandbox import ContainerSandboxBackend
ws_backend = ContainerSandboxBackend(
compute_client=compute_client,
user_id=str(user_id),
thread_id=thread_id,
input_objects=input_objects or [],
artifact_prefix=artifact_prefix,
default_output_paths=default_output_paths or ["/workspace/artifacts"],
max_runtime_seconds=max_runtime_seconds,
budget_snapshot=budget_snapshot or {},
execution_mode=execution_mode,
resource_class=resource_class,
backend_policy=backend_policy,
)
if _backend_ref is not None:
_backend_ref["backend"] = ws_backend
else:
# Fallback: file browsing/recovery only. Web shell execution is disabled.
from .paths import global_data_dir, user_data_dir
_uid = str(user_id)
if storage_backend == "file_gateway":
try:
from .runtime_integrations import resolve_user_storage_root
_user_root = resolve_user_storage_root(_uid)
if _user_root is None:
raise RuntimeError("No external user storage provider is configured")
_user_global = _user_root / "__global__"
except Exception:
# The mounted thread workspace is authoritative in
# FileGateway mode, even when the host integration provider
# is unavailable (for example in isolated tests/tools).
_user_root = Path(workspace_dir).resolve().parent
_user_global = _user_root / "__global__"
else:
_user_root = user_data_dir(_uid)
_user_global = global_data_dir(_uid)
_thread_root = Path(workspace_dir).resolve()
# Collect peer thread dirs (same user, other threads) as read-only sources
_peer_threads: list[str] = []
if _user_root.is_dir():
for d in sorted(_user_root.iterdir()):
if d.is_dir() and d.name != "__global__" and d.resolve() != _thread_root:
_peer_threads.append(str(d))
reason = "remote compute is unavailable"
if storage_backend not in ("nfs", "file_gateway"):
reason = f"storage backend {storage_backend!r} is unsupported for Web compute"
elif not compute_quota_ok:
reason = "compute quota preflight failed"
fallback_mode = (
compute_fallback_mode
or os.getenv("COMPUTE_FALLBACK_MODE", "local_sandbox")
or "local_sandbox"
).strip().lower()
if fallback_mode == "local_sandbox":
logger.warning(
"web_remote_compute_unavailable_using_local_sandbox thread=%s reason=%s",
thread_id,
reason,
)
ws_backend = MultiRootSandboxBackend(
write_root=str(_thread_root),
read_roots=_peer_threads,
global_root=str(_user_global),
virtual_mode=True,
timeout=max_runtime_seconds or 300,
)
else:
ws_backend = NonComputingFallbackBackend(
write_root=str(_thread_root),
read_roots=_peer_threads,
global_root=str(_user_global),
virtual_mode=True,
reason=reason,
)
else:
# CLI user: isolated sandbox (unchanged behavior)
ws_backend = CustomSandboxBackend(
root_dir=workspace_dir,
virtual_mode=True,
timeout=300,
)
sk_backend = MergedReadOnlyBackend(
primary_dir=_usr_skills_dir,
global_dir=_global_skills_dir,
secondary_dir=SKILLS_DIR,
)
# Memory always uses SHARED directory (not per-session) for cross-session persistence
mem_backend = FilesystemBackend(
root_dir=_mem_dir,
virtual_mode=True,
)
be = CompositeBackend(
default=ws_backend,
routes={
"/skills/": sk_backend,
"/memory/": mem_backend,
},
)
# Create chat model — use override if provided, otherwise use cached default
reasoning_effort = _resolve_reasoning_effort(reasoning_effort, cfg)
if model:
from .llm import get_chat_model
chat_model = get_chat_model(model=model, provider=provider)
else:
chat_model = _ensure_chat_model()
mw: list[AgentMiddleware] = [
create_context_editing_middleware(chat_model),
ContextOverflowMapperMiddleware(),
ToolErrorHandlerMiddleware(),
ToolResultLimiterMiddleware(mode_profile.max_tool_result_chars),
*_create_tool_selector_for_profile(
create_tool_selector_middleware,
mode_profile.tool_selector_threshold,
),
create_memory_middleware(
_mem_dir,
extraction_model=chat_model,
inject_memory=mode_profile.memory_injection_enabled,
max_injected_chars=mode_profile.memory_injection_max_chars,
),
]
if cfg.enable_ask_user and not cfg.auto_mode:
from .middleware.ask_user import AskUserMiddleware
mw.insert(0, AskUserMiddleware())
# Re-load MCP tools from current config (picks up /mcp add changes)
kwargs = load_mcp_and_build_kwargs(be, mw, chat_model=chat_model)
kwargs = _apply_conversation_profile(kwargs, mode_profile.mode)
# HITL: gate shell execution for user approval
_interrupt_on: dict[str, bool] | None = None
if not cfg.auto_approve:
_interrupt_on = {"execute": True}
mode_token = set_current_conversation_mode(mode_profile.mode)
try:
return create_deep_agent(
**kwargs,
checkpointer=checkpointer,
interrupt_on=_interrupt_on,
).with_config({"recursion_limit": 1000})
finally:
reset_current_conversation_mode(mode_token)
+79
View File
@@ -0,0 +1,79 @@
"""EvoScientist Agent - AI-powered research and code execution.
This package exposes a convenience API at the package root while keeping
imports lazy, so lightweight modules (for example config helpers) can be used
without importing heavy runtime dependencies.
"""
from __future__ import annotations
from importlib import import_module
from ._version import __version__
_EXPORTS: dict[str, tuple[str, str]] = {
# Agent graph (lazy to avoid expensive initialization at import time)
"EvoScientist_agent": (".EvoScientist", "EvoScientist_agent"),
"create_cli_agent": (".EvoScientist", "create_cli_agent"),
# Backends
"CustomSandboxBackend": (".backends", "CustomSandboxBackend"),
"ReadOnlyFilesystemBackend": (".backends", "ReadOnlyFilesystemBackend"),
# Configuration
"EvoScientistConfig": (".config", "EvoScientistConfig"),
"load_config": (".config", "load_config"),
"save_config": (".config", "save_config"),
"get_effective_config": (".config", "get_effective_config"),
"get_config_path": (".config", "get_config_path"),
# LLM
"get_chat_model": (".llm", "get_chat_model"),
"list_models": (".llm", "list_models"),
# Prompts
"get_system_prompt": (".prompts", "get_system_prompt"),
"RESEARCHER_INSTRUCTIONS": (".prompts", "RESEARCHER_INSTRUCTIONS"),
# Tools
"tavily_search": (".tools", "tavily_search"),
"web_search": (".tools", "web_search"),
"web_extract": (".tools", "web_extract"),
"web_crawl": (".tools", "web_crawl"),
"think_tool": (".tools", "think_tool"),
# Sessions
"get_checkpointer": (".sessions", "get_checkpointer"),
"generate_thread_id": (".sessions", "generate_thread_id"),
"list_threads": (".sessions", "list_threads"),
"delete_thread": (".sessions", "delete_thread"),
"get_storage_stats": (".sessions", "get_storage_stats"),
"get_aggregated_storage_stats": (".sessions", "get_aggregated_storage_stats"),
"list_all_session_db_paths": (".sessions", "list_all_session_db_paths"),
}
def __getattr__(name: str):
"""Lazily import and cache package-level attributes.
Args:
name: The attribute name to look up.
Returns:
The resolved attribute value.
Raises:
AttributeError: If the name is not in _EXPORTS.
"""
target = _EXPORTS.get(name)
if target is None:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
module_name, attr_name = target
module = import_module(module_name, package=__name__)
value = getattr(module, attr_name)
# Cache after first load to avoid repeated import lookups.
globals()[name] = value
return value
def __dir__() -> list[str]:
"""List available public attributes including lazy exports."""
return sorted(set(globals()) | set(_EXPORTS))
__all__ = list(_EXPORTS)
+5
View File
@@ -0,0 +1,5 @@
"""Enable `python -m EvoScientist` execution."""
from EvoScientist.cli import main
main()
+1
View File
@@ -0,0 +1 @@
__version__ = "0.1.19"
File diff suppressed because it is too large Load Diff
+348
View File
@@ -0,0 +1,348 @@
"""ccproxy lifecycle management for OAuth-based Anthropic access.
Provides functions to start/stop/health-check ccproxy, which allows
users with a Claude Pro/Max subscription to use EvoScientist without
a separate API key by reusing Claude Code's OAuth tokens.
ccproxy is invoked via subprocess (not Python imports) so the
``ccproxy-api`` package is truly optional at runtime.
"""
from __future__ import annotations
import logging
import os
import shutil
import subprocess
import time
from EvoScientist.config import EvoScientistConfig
logger = logging.getLogger(__name__)
# =============================================================================
# Availability & auth checks
# =============================================================================
def _ccproxy_exe() -> str | None:
"""Return the path to the ccproxy binary, or None if not found.
Checks PATH first, then the current Python environment's bin directory
(handles conda envs where newly installed binaries may not be visible
to shutil.which immediately after pip install).
"""
found = shutil.which("ccproxy")
if found:
return found
import sys as _sys
candidate = os.path.join(os.path.dirname(_sys.executable), "ccproxy")
if os.path.isfile(candidate) and os.access(candidate, os.X_OK):
return candidate
return None
def is_ccproxy_available() -> bool:
"""Check whether the ``ccproxy`` CLI binary is available."""
return _ccproxy_exe() is not None
def _is_editable_install() -> bool:
"""Return True if EvoScientist was installed in editable/development mode.
Checks all matching distributions because a stale ``.egg-info`` in the
project root can shadow the real ``dist-info`` in site-packages.
"""
try:
import importlib.metadata as _meta
import json
for dist in _meta.distributions():
name = dist.metadata.get("Name", "")
if name.lower() != "evoscientist":
continue
direct_url = dist.read_text("direct_url.json")
if direct_url is not None:
data = json.loads(direct_url)
if data.get("dir_info", {}).get("editable", False) is True:
return True
except Exception:
pass
return False
def _oauth_install_hint() -> str:
"""Return the appropriate install command depending on install method."""
if _is_editable_install():
return "uv sync --extra oauth or pip install -e '.[oauth]'"
return "pip install 'evoscientist[oauth]'"
def _summarize_auth_output(raw: str) -> str:
"""Extract key fields from ccproxy auth status output into a one-line summary.
Parses the Rich table output for Email, Subscription, and Status fields.
Returns e.g. ``"user@example.com (plus, active)"``.
Falls back to ``"Authenticated"`` if parsing fails.
"""
import re as _re
# Strip ANSI escape sequences
clean = _re.sub(r"\x1b\[[0-9;]*m", "", raw)
# Parse "Key<2+ spaces>Value" table rows, match exact key names
fields: dict[str, str] = {}
for line in clean.splitlines():
m = _re.match(r"\s*(.+?)\s{2,}(.+)", line)
if not m:
continue
key, val = m.group(1).strip(), m.group(2).strip()
if key in ("Email", "Subscription", "Subscription Status"):
fields[key.lower().replace(" ", "_")] = val
email = fields.get("email", "")
sub = fields.get("subscription", "")
status = fields.get("subscription_status", "")
if email:
detail = ", ".join(filter(None, [sub, status]))
return f"{email} ({detail})" if detail else email
return "Authenticated"
def check_ccproxy_auth(provider: str = "claude_api") -> tuple[bool, str]:
"""Check if ccproxy has valid OAuth credentials.
Args:
provider: ccproxy provider name ("claude_api" or "codex").
Returns:
(is_valid, message) tuple.
"""
try:
exe = _ccproxy_exe() or "ccproxy"
result = subprocess.run(
[exe, "auth", "status", provider],
capture_output=True,
text=True,
timeout=10,
)
import re as _re
raw = (result.stdout + result.stderr).strip()
clean = _re.sub(r"\x1b\[[0-9;]*m", "", raw)
# Filter out structlog warning/noise lines, keep only status lines
status_lines = [
line
for line in clean.splitlines()
if line.strip()
and not _re.match(r"\d{4}-\d{2}-\d{2}", line.strip())
and "warning" not in line.lower()
and "plugin" not in line.lower()
]
status_msg = " ".join(status_lines).strip()
# ccproxy auth status may exit 0 even when not authenticated —
# detect failure by checking output content
if result.returncode != 0 or "not authenticated" in clean.lower():
return False, status_msg or "Not authenticated"
summary = _summarize_auth_output(result.stdout)
return True, summary or "Authenticated"
except FileNotFoundError:
return False, "ccproxy not found"
except subprocess.TimeoutExpired:
return False, "Auth check timed out"
except Exception as exc:
return False, f"Auth check failed: {exc}"
# =============================================================================
# Process management
# =============================================================================
def is_ccproxy_running(port: int) -> bool:
"""Check if ccproxy is already serving on the given port."""
import httpx
try:
resp = httpx.get(f"http://127.0.0.1:{port}/health/live", timeout=2.0)
return resp.status_code == 200
except (httpx.ConnectError, httpx.TimeoutException, OSError):
return False
def start_ccproxy(port: int) -> subprocess.Popen:
"""Start ccproxy serve as a background process.
Args:
port: Port number for the proxy server.
Returns:
The Popen handle for the ccproxy process.
Raises:
RuntimeError: If ccproxy fails to become healthy within 30 seconds.
FileNotFoundError: If ccproxy binary is not found.
"""
exe = _ccproxy_exe() or "ccproxy"
proc = subprocess.Popen(
[exe, "serve", "--port", str(port)],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
# Wait for health (ccproxy can take up to ~11s on first start)
deadline = time.monotonic() + 30
while time.monotonic() < deadline:
if proc.poll() is not None:
raise RuntimeError(
f"ccproxy exited immediately with code {proc.returncode}"
)
if is_ccproxy_running(port):
return proc
time.sleep(0.3)
# Timed out — clean up
proc.terminate()
try:
proc.wait(timeout=3)
except subprocess.TimeoutExpired:
proc.kill()
raise RuntimeError("ccproxy did not become healthy within 30 seconds")
def stop_ccproxy(proc: subprocess.Popen | None) -> None:
"""Gracefully stop a ccproxy process.
Safe to call with None (no-op).
"""
if proc is None:
return
try:
proc.terminate()
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait(timeout=2)
except Exception:
pass
def ensure_ccproxy(port: int) -> subprocess.Popen | None:
"""Ensure ccproxy is running — reuse existing or start new.
Returns:
Popen handle if we started a new process, None if already running.
"""
if is_ccproxy_running(port):
logger.debug("ccproxy already running on port %d", port)
return None
return start_ccproxy(port)
# =============================================================================
# Environment setup
# =============================================================================
def setup_ccproxy_env(port: int) -> None:
"""Set environment variables for Anthropic ccproxy routing.
Force-sets ``ANTHROPIC_BASE_URL`` and ``ANTHROPIC_API_KEY`` so that
downstream LangChain/Anthropic clients route through ccproxy.
Always overrides existing values — when this function is called,
we've decided to use ccproxy, so env must point to it.
"""
os.environ["ANTHROPIC_BASE_URL"] = f"http://127.0.0.1:{port}/claude"
os.environ["ANTHROPIC_API_KEY"] = "ccproxy-oauth"
def setup_codex_env(port: int) -> None:
"""Set environment variables for OpenAI/Codex ccproxy routing.
Force-sets ``OPENAI_BASE_URL`` and ``OPENAI_API_KEY`` so that
downstream LangChain/OpenAI clients route through ccproxy's Codex
endpoint.
Always overrides existing values — when this function is called,
we've decided to use ccproxy, so env must point to it.
"""
os.environ["OPENAI_BASE_URL"] = f"http://127.0.0.1:{port}/codex/v1"
os.environ["OPENAI_API_KEY"] = "ccproxy-oauth"
# =============================================================================
# High-level orchestration
# =============================================================================
def maybe_start_ccproxy(config: EvoScientistConfig) -> subprocess.Popen | None:
"""High-level: conditionally start ccproxy based on config.
Checks ``config.anthropic_auth_mode`` and ``config.openai_auth_mode``:
- ``oauth``: ccproxy must work — raises on failure.
- ``api_key``: no-op for that provider.
When either provider uses OAuth, ccproxy is started (single process
serves both providers). Environment variables are set for each
provider that uses OAuth.
Args:
config: An ``EvoScientistConfig`` instance.
Returns:
Popen handle if we started ccproxy, None otherwise.
"""
anthropic_oauth = getattr(config, "anthropic_auth_mode", "api_key") == "oauth"
openai_oauth = getattr(config, "openai_auth_mode", "api_key") == "oauth"
if not anthropic_oauth and not openai_oauth:
return None
if not is_ccproxy_available():
raise RuntimeError(
"ccproxy is required for OAuth mode but not found. "
f"Install it with: {_oauth_install_hint()}"
)
# Check auth for each provider that uses OAuth
if anthropic_oauth:
authed, msg = check_ccproxy_auth("claude_api")
if not authed:
raise RuntimeError(
f"ccproxy Anthropic OAuth not authenticated: {msg}\n"
"Run: ccproxy auth login claude_api"
)
if openai_oauth:
authed, msg = check_ccproxy_auth("codex")
if not authed:
raise RuntimeError(
f"ccproxy Codex OAuth not authenticated: {msg}\n"
"Run: ccproxy auth login codex"
)
port = config.ccproxy_port
if not (1 <= port <= 65535):
raise ValueError(f"Invalid ccproxy port: {port}. Must be between 1 and 65535.")
# Start ccproxy (single process serves both providers)
proc = ensure_ccproxy(port)
# Set environment for each OAuth provider
if anthropic_oauth:
setup_ccproxy_env(port)
if openai_oauth:
setup_codex_env(port)
if proc:
logger.info("Started ccproxy on port %d", port)
else:
logger.info("Reusing existing ccproxy on port %d", port)
return proc
+928
View File
@@ -0,0 +1,928 @@
# Channels
EvoScientist provides unified integration with 10 messaging platforms. This document covers the architecture overview, message processing pipeline, capability matrix, security model, deployment guides, and troubleshooting.
Configuration file: `~/.config/ai4scientist/settings.yaml` (or use environment variables with the `EVOSCIENTIST_` prefix).
## Table of Contents
- [Architecture](#architecture)
- [Message Processing Pipeline](#message-processing-pipeline)
- [Middleware Pipeline](#middleware-pipeline)
- [Capability Matrix](#capability-matrix)
- [Security and Access Control](#security-and-access-control)
- [Quick Start](#quick-start)
- [Channel Deployment Guides](#channel-deployment-guides)
- [Telegram](#telegram) | [Discord](#discord) | [Slack](#slack) | [Feishu (Lark)](#feishu-lark) | [WeChat](#wechat)
- [DingTalk](#dingtalk) | [QQ](#qq) | [Signal](#signal) | [Email](#email) | [iMessage](#imessage)
- [Running Multiple Channels](#running-multiple-channels)
- [Docker Deployment](#docker-deployment)
- [Troubleshooting](#troubleshooting)
## Architecture
```
┌─────────────────────────────────────────────┐
│ Messaging Platforms │
│ │
│ ┌────────┐ ┌────────┐ ┌────────┐ │
│ │Telegram│ │Discord │ │ Slack │ ...x10 │
│ └───┬────┘ └───┬────┘ └───┬────┘ │
└──────┼──────────┼──────────┼───────────────┘
│ │ │
┌──────┴──────────┴──────────┴───────────────┐
│ Inbound Middleware │
│ │
│ Dedup → AllowList → Pairing → GroupHist │
│ → Mention │
└──────────────────┬─────────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ MessageBus │
│ │
│ inbound queue ──► outbound queue │
│ (asyncio.Queue, capacity 5000) │
└──────────┬───────────────┬───────────┘
│ │
▼ ▼
┌──────────────────┐ ┌─────────────────┐
│ InboundConsumer │ │ Dispatcher │
│ │ │ │
│ Worker pool (8) │ │ Routes replies │
│ Per-chat locks │ │ to origin │
│ Session dedup │ │ channel │
│ Timeout handling │ │ │
│ │ │ └─────────────────┘
│ ▼ │
│ Agent Core │
└──────────────────┘
```
### Core Modules
| Module | Responsibility |
|--------|---------------|
| `base.py` | Abstract `Channel` base class — readiness checks, retry strategy, mention stripping, format fallback, media handling, debounce, send locks |
| `capabilities.py` | `ChannelCapabilities` frozen dataclass — each channel declares features, framework adapts automatically |
| `plugin.py` | `ChannelPlugin` base with adapter slots — `ConfigAdapter`, `SecurityAdapter`, `GroupAdapter`, `MentionAdapter`, `OutboundAdapter`, `ThreadingAdapter`, etc. |
| `mixins.py` | Reusable async patterns: `WebhookMixin` (aiohttp server + httpx client), `WebSocketMixin` (connect/reconnect/heartbeat), `PollingMixin` (async polling loop), `TokenMixin` (OAuth token auto-refresh) |
| `config.py` | `BaseChannelConfig` — shared config fields (allowed_senders, proxy, text_chunk_limit, etc.) + `SingleAccountConfigAdapter` / `MultiAccountConfigAdapter` |
| `bus/` | `MessageBus` async event bus with `InboundMessage` / `OutboundMessage` dataclasses, decoupling channels from agent core |
| `channel_manager.py` | `ChannelManager` — lifecycle management (start/stop), health monitoring, channel registry, account management, outbound dispatch |
| `consumer.py` | `InboundConsumer` — worker pool, per-chat serial locks, session deduplication, timeout handling |
| `retry.py` | `RetryConfig` — exponential backoff retry with per-channel presets (attempts, min/max delay, jitter) |
| `formatter.py` | `UnifiedFormatter` — Markdown to platform-specific format conversion (HTML, Slack mrkdwn, Discord, plain text) |
| `standalone.py` | Headless channel runner (`run_standalone`) for running channels without the CLI |
## Message Processing Pipeline
### Inbound (User Message → Agent)
```
1. Platform SDK/Webhook receives raw message
│
2. Channel._on_message() parses into RawIncoming
│
3. Channel._enqueue_raw() runs middleware pipeline:
├── DedupMiddleware — drop duplicates (LRU cache, 60s TTL)
├── AllowListMiddleware — enforce sender/channel restrictions
├── PairingMiddleware — handle DM pairing flow (if dm_policy="pairing")
├── GroupHistoryMiddleware — buffer group context for injection
└── MentionGatingMiddleware — filter by @mention policy in groups
│
4. InboundMessage queued on Channel._queue
│
5. Channel.run() → receive() → queue_message() with debounce
│
│ (500ms debounce window: rapid messages from same sender merged)
│
6. MessageBus.publish_inbound()
│
7. InboundConsumer acquires per-chat lock → invokes Agent
│
8. Agent response → OutboundMessage → MessageBus.publish_outbound()
```
### Outbound (Agent Response → User)
```
1. OutboundMessage arrives on MessageBus outbound queue
│
2. Dispatcher routes to origin channel by name
│
3. Channel.send() processes the response:
├── Stop typing indicator
├── Format text (Markdown → platform format)
├── Chunk text to platform limit (code-block-aware splitting)
├── Send each chunk via _send_chunk() with format fallback
├── Send media attachments via _send_media_impl()
└── Retry on transient errors (exponential backoff)
```
### Text Chunking
Long responses are split intelligently with this priority:
1. Markdown code block fence boundaries
2. Double newlines (paragraph breaks)
3. Single newlines
4. Space characters
5. Hard cut at limit (last resort)
Code blocks are never split mid-block when possible. Each chunk is sent as a separate message.
## Middleware Pipeline
Middleware runs sequentially on each inbound message. Each middleware can pass, modify, or drop the message.
### 1. DedupMiddleware
Prevents duplicate message processing using a bounded LRU cache with TTL.
- Cache size: 1000 entries (configurable)
- TTL: 60 seconds
- Key: `message_id` from the platform
- Messages with the same ID within the TTL window are silently dropped
### 2. AllowListMiddleware
Enforces sender and channel restrictions based on the `dm_policy` config.
| Policy | Behavior |
|--------|----------|
| `"open"` | Accept messages from anyone |
| `"allowlist"` | Only accept from `allowed_senders` / `allowed_channels` |
| `"pairing"` | Require DM pairing before accepting (see PairingMiddleware) |
When `allowed_senders` is set (non-empty), only messages from listed sender IDs pass through. Same for `allowed_channels`.
### 3. PairingMiddleware
Handles an interactive DM pairing flow for the `"pairing"` dm_policy.
- First message from an unknown sender triggers a pairing request
- The sender must provide a valid pairing code
- Once paired, the sender is added to the allowlist for future messages
### 4. GroupHistoryMiddleware
Buffers recent group chat messages to provide conversation context.
- Only active when `capabilities.groups = True`
- Maintains a per-chat rolling buffer (default: 50 messages, 5-minute max age)
- When the bot is mentioned in a group, recent history is injected into the message metadata so the agent can see prior context
- Non-mentioned group messages are buffered but not forwarded (see MentionGating)
### 5. MentionGatingMiddleware
Controls whether the bot responds in group chats.
| `require_mention` | Behavior |
|-------------------|----------|
| `True` / `"group"` | Only respond to @mentions in groups; always respond in DMs |
| `False` / `"none"` | Respond to all messages in all contexts |
| `"always"` | Require @mention even in DMs |
Default: `"group"` — the bot ignores group messages unless explicitly @mentioned.
Mention detection is platform-specific:
- **Telegram**: checks for `@bot_username` in text
- **Discord**: checks `message.mentions` for bot user
- **Slack**: handled via separate `app_mention` event type
- **Feishu**: checks `mentions` array in event payload
- **DingTalk**: checks `isInAtList` flag or `atUsers` array
- **WeChat (WeCom)**: checks `AtUserList` XML field
## Capability Matrix
| Channel | Format | Max Len | Media | Voice | Sticker | Location | Video | Typing | Reaction | Thread | Group | @Mention | No Public IP | Token Refresh | Proxy | Allowlist |
|:--------|:------:|:-------:|:-----:|:-----:|:-------:|:--------:|:-----:|:------:|:--------:|:------:|:-----:|:--------:|:------------:|:-------------:|:-----:|:---------:|
| Telegram | HTML | 4000 | S/R | R | R | R | | 4s | emoji | | G | @ | yes | | yes | yes |
| Discord | Discord | 2000 | S/R | | | | | 8s | emoji | yes | G | @ | yes | | yes | yes |
| Slack | Mrkdwn | 4000 | S/R | | | | | post | emoji | yes | G | @ | yes | | yes | yes |
| Feishu | Post | 4096 | S/R | R | R | | | | emoji | | G | @ | ws mode | 2h | yes | yes |
| WeChat | MD | 4096 | S/R | R | | R | R | recall | | | G | @ | no | 2h | yes | yes |
| DingTalk | MD | 4096 | S/R | R | | | R | | | | G | @ | yes | 2h | yes | yes |
| QQ | MD/plain | 4096 | S/R | | | | | | | | G | @ | yes | | | yes |
| Signal | Plain | 4096 | S/R | R | | | | api | emoji | | G | UUID | yes | | | yes |
| iMessage | Plain | - | S/R | R | | | | | | | G | | yes | | | yes |
| Email | HTML | - | S/R | | | | | | | | | | yes | | | yes |
Legend: **S** = send, **R** = receive, **G** = group chat, **@** = @mention detection, **-** = no practical limit
### Connection Types
| Channel | Transport | Connection Mode | Default Port |
|-------------|-----------|----------------------------------------|:------------:|
| Telegram | HTTPS | Long polling (`getUpdates`) | -- |
| Discord | WebSocket | Gateway events (`discord.py`) | -- |
| Slack | WebSocket | Socket Mode (`slack-sdk`) | -- |
| Feishu | HTTP/WS | Webhook or WebSocket long connection | 9000/-- |
| WeChat | HTTP | Webhook `POST /wechat/callback` | 9001 |
| DingTalk | WebSocket | Stream Mode (DingTalk gateway) | -- |
| QQ | WebSocket | Bot Gateway (`qq-botpy`) | -- |
| Signal | TCP | JSON-RPC (`signal-cli` daemon) | 7583 |
| iMessage | stdio | JSON-RPC (`imsg` CLI) | -- |
| Email | TCP | IMAP polling + SMTP send | 993/587 |
> **"--"** means no listening port is required -- no public IP or port forwarding needed.
### Format Conversion
The `UnifiedFormatter` converts Markdown output from the agent into platform-native formats:
| Target Format | Conversion |
|:-------------|:-----------|
| HTML (Telegram, Email) | `**bold**` → `<b>bold</b>`, `` `code` `` → `<code>code</code>`, code blocks → `<pre>`, special chars escaped |
| Slack mrkdwn | `**bold**` → `*bold*`, `_italic_` → `_italic_`, code blocks preserved, `<>&` escaped |
| Discord Markdown | Mostly passthrough, minor adjustments for Discord-specific rendering |
| Feishu Post | Markdown → Feishu rich text JSON (code blocks, bold, italic, strikethrough, links, headings, quotes, lists) |
| Plain text | All formatting stripped, structure preserved via indentation |
## Security and Access Control
### Sender Allowlist
Every channel supports `allowed_senders` to restrict who can interact with the bot:
```yaml
telegram_allowed_senders: "123456789,987654321" # Telegram user IDs
discord_allowed_senders: "111222333444555666" # Discord user IDs
slack_allowed_senders: "U0123ABCDEF" # Slack Member IDs
feishu_allowed_senders: "ou_xxxxxxxxxxxx" # Feishu open_ids
signal_allowed_senders: "+1234567890" # Phone numbers
email_allowed_senders: "alice@example.com" # Email addresses
imessage_allowed_senders: "+1234567890,user@icloud.com" # Phone or email
```
When `allowed_senders` is empty, the channel accepts messages from anyone. **For production deployments, always set an allowlist.**
### Channel Allowlist
For platforms with multiple channels/groups (Discord, Slack), restrict which channels the bot operates in:
```yaml
discord_allowed_channels: "111222333444555666,777888999000111222"
slack_allowed_channels: "C0123ABCDEF,C0456GHIJKL"
```
### Token and Secret Handling
- API tokens are stored in the config file or environment variables, never logged at INFO level
- Discord logs only the first 8 and last 4 characters of the bot token for debugging
- WeChat/Feishu tokens are auto-refreshed before expiry (5-minute margin on 2-hour TTL)
- Webhook signature verification is enforced when `token`/`encoding_aes_key` is configured (WeChat, Feishu)
### Group Chat Behavior
By default, the bot only responds in group chats when explicitly @mentioned. This prevents the bot from responding to every message in a busy group. Configure via:
```yaml
# Default: only respond when mentioned in groups
channel_require_mention: "group"
# Respond to all messages (including groups)
channel_require_mention: "none"
```
## Quick Start
### 1. Install channel dependencies
```bash
pip install evoscientist[telegram]
# Available extras: telegram, discord, slack, feishu, wechat,
# dingtalk, qq, email, signal
# iMessage requires no extra Python dependencies
```
### 2. Configure
```bash
# Option A: Interactive wizard
EvoSci onboard
# Option B: CLI commands
EvoSci config set channel_enabled telegram
EvoSci config set telegram_bot_token "123456:ABC-xxx"
# Option C: Environment variables (EVOSCIENTIST_ prefix, uppercase)
export EVOSCIENTIST_CHANNEL_ENABLED=telegram
export EVOSCIENTIST_TELEGRAM_BOT_TOKEN="123456:ABC-xxx"
```
### 3. Start
```bash
EvoSci serve # Start agent + all enabled channels
# or
EvoSci channel start # Standalone channel mode (message loop only)
```
### 4. Health check
```bash
curl http://localhost:8080/healthz
```
```json
{
"status": "healthy",
"channels": { "enabled": ["telegram"], "running": ["telegram"] }
}
```
---
## Channel Deployment Guides
---
### Telegram
**Install:** `pip install evoscientist[telegram]`
**Prerequisites:**
1. Search for [@BotFather](https://t.me/BotFather) in Telegram, send `/newbot`, and follow the prompts to create a bot.
2. BotFather will return a Bot Token (format: `123456789:ABCdefGHI...`) -- save it securely.
3. Get your user ID: send any message to [@userinfobot](https://t.me/userinfobot), it will reply with your numeric ID.
4. (Optional) For group use: add the bot to a group, then in BotFather send `/setprivacy` -> `Disable` so the bot can read group messages.
**Configuration:**
```yaml
channel_enabled: "telegram"
telegram_bot_token: "123456789:ABCdefGHIjklMNOpqrSTUvwxYZ"
telegram_allowed_senders: "" # Comma-separated user IDs; empty = no restriction
telegram_proxy: "" # Optional HTTPS proxy (e.g. http://proxy:8080)
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `telegram_bot_token` | `str` | `""` | **Required.** Bot API Token from BotFather |
| `telegram_allowed_senders` | `str` | `""` | Comma-separated user IDs, empty = allow all |
| `telegram_proxy` | `str` | `""` | HTTPS proxy URL |
**Env vars:** `EVOSCIENTIST_TELEGRAM_BOT_TOKEN`, `EVOSCIENTIST_TELEGRAM_ALLOWED_SENDERS`, `EVOSCIENTIST_TELEGRAM_PROXY`
**Technical details:** Long polling mode, `drop_pending_updates=True` on startup to skip backlog. Markdown to Telegram HTML auto-conversion (bold, italic, strikethrough, links, code blocks, headings, lists). Falls back to plain text on HTML parse failure. Media routed by extension to `send_photo`/`send_video`/`send_audio`/`send_document`. In groups, only responds when @mentioned; auto-strips @mention. Typing indicator refreshes every 4s. ACK reaction (eyes emoji) on message receipt, removed after reply. Retry: 3 attempts, min delay 0.4s, parse errors not retried. Text chunk limit: 4000 chars.
---
### Discord
**Install:** `pip install evoscientist[discord]`
**Prerequisites:**
1. Go to [Discord Developer Portal](https://discord.com/developers/applications) -> New Application -> enter a name.
2. Left menu **Bot** -> Reset Token -> copy the Bot Token.
3. Under **Privileged Gateway Intents**, enable **Message Content Intent** (required to read message content).
4. Left menu **OAuth2** -> URL Generator:
- Scopes: check `bot`
- Bot Permissions: check `Send Messages`, `Read Message History`, `Attach Files`, `Add Reactions`
- Copy the generated URL, open in browser, select a server to invite the bot.
5. Get user ID: Discord Settings -> Advanced -> enable Developer Mode -> right-click username -> Copy User ID.
**Configuration:**
```yaml
channel_enabled: "discord"
discord_bot_token: "MTIzNDU2Nzg5.xxxx.xxxxx"
discord_allowed_senders: "" # Comma-separated user IDs
discord_allowed_channels: "" # Comma-separated channel IDs
discord_proxy: ""
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `discord_bot_token` | `str` | `""` | **Required.** Bot Token |
| `discord_allowed_senders` | `str` | `""` | Comma-separated user IDs, empty = allow all |
| `discord_allowed_channels` | `str` | `""` | Comma-separated channel IDs, empty = allow all |
| `discord_proxy` | `str` | `""` | HTTPS proxy URL |
**Env vars:** `EVOSCIENTIST_DISCORD_BOT_TOKEN`, `EVOSCIENTIST_DISCORD_ALLOWED_SENDERS`, `EVOSCIENTIST_DISCORD_ALLOWED_CHANNELS`, `EVOSCIENTIST_DISCORD_PROXY`
**Technical details:** WebSocket Gateway (`discord.py`). In server channels, only responds when @mentioned; DMs respond directly. Thread-aware: messages in threads are tracked with `parent_channel_id` and `thread_id`. Replies via `MessageReference`. Message cache (200 entries) for ACK emoji reactions. Attachment download (max 20 MB) with safe filename sanitization. Media sent via `discord.File`. Typing indicator refreshes every 8s. Retry: 3 attempts, parses `Retry-After` header for 429s. Text chunk limit: 2000 chars.
---
### Slack
**Install:** `pip install evoscientist[slack]`
**Prerequisites:**
1. Go to [Slack API](https://api.slack.com/apps) -> Create New App -> From scratch -> select workspace.
2. Left menu **Socket Mode** -> enable -> Generate App-Level Token, scope `connections:write` -> copy App Token (`xapp-...`).
3. Left menu **OAuth & Permissions** -> add Bot Token Scopes:
- `chat:write`, `channels:history`, `groups:history`, `im:history`, `files:read`, `files:write`, `reactions:write`
4. Click **Install to Workspace** -> copy Bot User OAuth Token (`xoxb-...`).
5. Left menu **Event Subscriptions** -> enable -> Subscribe to bot events: `message.channels`, `message.groups`, `message.im`, `app_mention`.
6. Get Member ID: click user avatar -> profile -> **...** -> Copy member ID.
**Configuration:**
```yaml
channel_enabled: "slack"
slack_bot_token: "xoxb-xxxx-xxxx-xxxx"
slack_app_token: "xapp-1-xxxx-xxxx"
slack_allowed_senders: "" # Member ID (U...)
slack_allowed_channels: "" # Channel ID (C...)
slack_proxy: ""
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `slack_bot_token` | `str` | `""` | **Required.** Bot User OAuth Token (`xoxb-`) |
| `slack_app_token` | `str` | `""` | **Required.** Socket Mode App Token (`xapp-`) |
| `slack_allowed_senders` | `str` | `""` | Comma-separated Member IDs |
| `slack_allowed_channels` | `str` | `""` | Comma-separated Channel IDs |
| `slack_proxy` | `str` | `""` | HTTPS proxy URL |
**Env vars:** `EVOSCIENTIST_SLACK_BOT_TOKEN`, `EVOSCIENTIST_SLACK_APP_TOKEN`, `EVOSCIENTIST_SLACK_ALLOWED_SENDERS`, `EVOSCIENTIST_SLACK_ALLOWED_CHANNELS`, `EVOSCIENTIST_SLACK_PROXY`
**Technical details:** Socket Mode (no public URL needed). Markdown to mrkdwn conversion. DMs respond directly; channels respond to `app_mention` events. Thread replies via `thread_ts` -- all replies are threaded to the original message. Typing indicator approximated by posting/deleting a "..." message (Slack has no bot typing API). ACK reaction (eyes emoji) on message receipt. Attachments downloaded with Bearer auth. Media sent via `files_upload_v2`. Runs `auth_test()` on startup to verify credentials and cache bot user ID. Retry: 3 attempts, exponential backoff + jitter. Text chunk limit: 4000 chars.
---
### Feishu (Lark)
**Install:** `pip install evoscientist[feishu]`
**Prerequisites:**
1. Go to [Feishu Open Platform](https://open.feishu.cn/app) (international: [Lark Developer](https://open.larksuite.com/app)) -> create a custom app.
2. Copy the **App ID** and **App Secret**.
3. Left menu **Event Subscriptions**:
- **Webhook mode**: set request URL to `http://your-host:9000/webhook/event` -> copy **Verification Token** and **Encrypt Key**.
- **WebSocket mode**: select **长连接** (Long Connection) as the subscription method. No URL needed.
4. Add event: `im.message.receive_v1` (receive messages).
5. Left menu **Permissions** -> enable `im:message:send_as_bot`.
6. Create a version and publish.
> **Webhook mode** requires a publicly reachable URL. For local dev, use `ngrok http 9000` or [natapp](https://natapp.cn/) (recommended for China).
#### Subscription Modes
Feishu supports two subscription modes:
| Mode | Transport | Public IP? | Best For |
|------|-----------|:----------:|----------|
| `webhook` (default) | HTTP POST callback | Yes | Servers with public IP / cloud deployment |
| `websocket` | WebSocket long connection | **No** | Local dev, behind NAT/firewall, China (no ngrok needed) |
**WebSocket mode** uses the official `lark-oapi` SDK to maintain an outbound WebSocket connection to Feishu servers. No public IP, port forwarding, or tunnel is required.
To use WebSocket mode:
```bash
# Install the SDK
pip install 'evoscientist[feishu]'
# Via config file
feishu_subscription_mode: "websocket"
# Via CLI (standalone)
python -m EvoScientist.channels.feishu.serve --app-id ID --app-secret SECRET --mode websocket
```
> **Note:** In WebSocket mode, `feishu_verification_token`, `feishu_encrypt_key`, and `feishu_webhook_port` are not used — the SDK handles authentication and encryption internally.
**Configuration:**
```yaml
channel_enabled: "feishu"
feishu_app_id: "cli_xxxxxxx"
feishu_app_secret: "xxxxxxxxxxxxxxxxxx"
feishu_subscription_mode: "webhook" # or "websocket"
feishu_webhook_port: 9000
feishu_allowed_senders: "" # open_id
feishu_domain: "https://open.feishu.cn"
feishu_proxy: ""
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `feishu_app_id` | `str` | `""` | **Required.** App ID |
| `feishu_app_secret` | `str` | `""` | **Required.** App Secret |
| `feishu_webhook_port` | `int` | `9000` | Webhook HTTP port |
| `feishu_allowed_senders` | `str` | `""` | Comma-separated open_ids |
| `feishu_domain` | `str` | `"https://open.feishu.cn"` | API domain (use `https://open.larksuite.com` for Lark) |
| `feishu_proxy` | `str` | `""` | HTTPS proxy URL |
| `feishu_subscription_mode` | `str` | `"webhook"` | `"webhook"` or `"websocket"` (WebSocket long connection, no public IP) |
**Env vars:** `EVOSCIENTIST_FEISHU_APP_ID`, `EVOSCIENTIST_FEISHU_APP_SECRET`, `EVOSCIENTIST_FEISHU_WEBHOOK_PORT`, `EVOSCIENTIST_FEISHU_DOMAIN`
**Technical details:** Webhook on `POST /webhook/event` with URL verification challenge-response. Supports both v1 (legacy) and v2 event schemas. Optional AES-256-CBC event decryption (when `encrypt_key` configured). `tenant_access_token` auto-refresh (2h TTL, refreshes 5 min before expiry). Markdown to Feishu Post rich text conversion (code blocks, bold, italic, strikethrough, links, headings, quotes, ordered/unordered lists). Plain text fallback. Group @mention filtering with mention key caching. Media: images via `/im/v1/images`, files via `/im/v1/files`. Replies via `/messages/{id}/reply` API. ACK reaction via `/messages/{id}/reactions`. Retry: 3 attempts, rate limit delay 2.0s, matches `99991400`/`rate limit`. Non-retryable: permission denied (`99991401`), invalid credentials. Text chunk limit: 4096 chars.
---
### WeChat
**Install:** `pip install evoscientist[wechat]`
Two backends supported: **WeCom** (recommended, free, no certification needed) and **WeChat Official Account** (requires verified service account).
#### WeCom
**Prerequisites:**
1. Log in to [WeCom Admin Console](https://work.weixin.qq.com) -> App Management -> create a custom app.
2. Copy the **Corp ID**, **AgentId**, and **Secret**.
3. In app details -> Receive Messages -> Set API Receive -> URL: `http://your-host:9001/wechat/callback` -> copy **Token** and **EncodingAESKey**.
4. In app details -> **Trusted IP** -> add your server's public IP address. Without this, all API calls will fail with error `60020`.
```yaml
channel_enabled: "wechat"
wechat_backend: "wecom"
wechat_webhook_port: 9001
wechat_wecom_corp_id: "ww..."
wechat_wecom_agent_id: "1000002"
wechat_wecom_secret: "xxxxxxxxxxxxxxxxxx"
wechat_wecom_token: "xxxxxxxxxxxxxxxxxx"
wechat_wecom_encoding_aes_key: "xxxxxxxxxxxxxxxxxx"
wechat_allowed_senders: ""
wechat_proxy: ""
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `wechat_backend` | `str` | `"wecom"` | `"wecom"` or `"wechatmp"` |
| `wechat_webhook_port` | `int` | `9001` | Callback HTTP port |
| `wechat_wecom_corp_id` | `str` | `""` | **Required (WeCom).** Corp ID |
| `wechat_wecom_agent_id` | `str` | `""` | **Required (WeCom).** App AgentId |
| `wechat_wecom_secret` | `str` | `""` | **Required (WeCom).** App Secret |
| `wechat_wecom_token` | `str` | `""` | **Required (WeCom).** Callback Token |
| `wechat_wecom_encoding_aes_key` | `str` | `""` | **Required (WeCom).** Callback EncodingAESKey |
#### WeChat Official Account
**Prerequisites:**
1. Log in to [WeChat Official Account Platform](https://mp.weixin.qq.com) -> Settings & Development -> Basic Configuration.
2. Copy the **AppID** and **AppSecret**.
3. Server Configuration -> URL: `http://your-host:9001/wechat/callback` -> set **Token** and **EncodingAESKey**.
```yaml
wechat_backend: "wechatmp"
wechat_mp_app_id: "wx..."
wechat_mp_app_secret: "xxxxxxxxxxxxxxxxxx"
wechat_mp_token: "xxxxxxxxxxxxxxxxxx"
wechat_mp_encoding_aes_key: "xxxxxxxxxxxxxxxxxx"
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `wechat_mp_app_id` | `str` | `""` | **Required (MP).** AppID |
| `wechat_mp_app_secret` | `str` | `""` | **Required (MP).** AppSecret |
| `wechat_mp_token` | `str` | `""` | **Required (MP).** Server Token |
| `wechat_mp_encoding_aes_key` | `str` | `""` | **Required (MP).** Server EncodingAESKey |
**Technical details:** Webhook HTTP server for inbound (XML message parsing). GET callback for URL verification (SHA1 signature check). POST callback for message handling -- returns `"success"` within 5s and processes asynchronously. Optional AES encryption/decryption via `WeChatCrypto`. `access_token` auto-refresh (2h TTL, 5-min margin). Token-expired errors (40014, 42001) trigger automatic retry with refreshed token. WeCom supports Markdown message format with plain text fallback; Official Account uses plain text only (customer service API). WeCom group messages sent via `/appchat/send` endpoint (group IDs start with `wr`). Typing indicator approximated by posting/recalling a "..." message (WeCom only). Supports text, image, voice, video, location, file, and link message types. Media upload via `/media/upload`. Text chunk limit: 4096 chars.
---
### DingTalk
**Install:** `pip install evoscientist[dingtalk]`
**Prerequisites:**
1. Go to [DingTalk Open Platform](https://open-dev.dingtalk.com) -> App Development -> create a bot app.
2. Copy the **AppKey** (Client ID) and **AppSecret** (Client Secret).
3. Enable **Stream Mode** in the app configuration -- no public IP needed.
4. Publish the app and add the bot to a group, or test via direct message.
**Configuration:**
```yaml
channel_enabled: "dingtalk"
dingtalk_client_id: "ding..."
dingtalk_client_secret: "xxxxxxxxxxxxxxxxxx"
dingtalk_allowed_senders: ""
dingtalk_proxy: ""
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `dingtalk_client_id` | `str` | `""` | **Required.** AppKey |
| `dingtalk_client_secret` | `str` | `""` | **Required.** AppSecret |
| `dingtalk_allowed_senders` | `str` | `""` | Comma-separated user IDs |
| `dingtalk_proxy` | `str` | `""` | HTTPS proxy URL |
**Env vars:** `EVOSCIENTIST_DINGTALK_CLIENT_ID`, `EVOSCIENTIST_DINGTALK_CLIENT_SECRET`
**Technical details:** Stream Mode via WebSocket -- connects to DingTalk gateway (`/v1.0/gateway/connections/open`) with automatic ticket-based auth. Ping/pong heartbeat with system topic handling. Message ACK via JSON response. `accessToken` auto-refresh. Sends via robot `oToMessages/batchSend` API in Markdown format (`sampleMarkdown`). Image uploads via `/media/upload` API with `sampleImageMsg`. Group @mention detection via `isInAtList` flag with `atUsers` array fallback. `downloadCode` resolution via `/robot/messageFiles/download` API for file/image/video/audio attachments. Auth errors (`invalidauthentication`/`forbidden`/`40014`) not retried. Text chunk limit: 4096 chars.
---
### QQ
**Install:** `pip install evoscientist[qq]`
**Prerequisites:**
1. Go to [QQ Open Platform](https://q.qq.com) -> create a bot application.
2. Complete developer verification, create a sandbox or production bot.
3. Copy the **AppID** and **AppSecret**.
4. Search for and add the bot as a friend in QQ, or add it to a group.
**Configuration:**
```yaml
channel_enabled: "qq"
qq_app_id: "xxxxxxxxxx"
qq_app_secret: "xxxxxxxxxxxxxxxxxx"
qq_allowed_senders: ""
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `qq_app_id` | `str` | `""` | **Required.** AppID |
| `qq_app_secret` | `str` | `""` | **Required.** AppSecret |
| `qq_allowed_senders` | `str` | `""` | Comma-separated user IDs |
**Env vars:** `EVOSCIENTIST_QQ_APP_ID`, `EVOSCIENTIST_QQ_APP_SECRET`
**Technical details:** Uses `qq-botpy` SDK via WebSocket to connect to QQ Bot Gateway. Supports C2C (direct) and group messages. Outbound replies prefer native QQ markdown messages (`msg_type=2`) so headings/lists/code fences keep their structure; when the SDK/API rejects markdown, EvoScientist falls back to plain text with Markdown stripped but line structure preserved. Message deduplication (1000-entry LRU cache). Group @mention filtering (strips first `@bot`). Intents: `public_messages=True`, `direct_message=True`. Text chunk limit: 4096 chars.
---
### Signal
**Install:** `pip install evoscientist[signal]` (also requires [signal-cli](https://github.com/AsamK/signal-cli) installed separately)
**Prerequisites:**
1. Install signal-cli: see [signal-cli installation guide](https://github.com/AsamK/signal-cli#installation).
2. Register or link a phone number:
- Register: `signal-cli -u +1234567890 register`, then `signal-cli -u +1234567890 verify CODE`
- Link existing device: `signal-cli link -n "EvoScientist"`
3. EvoScientist will auto-start the signal-cli daemon if it's not already running.
**Configuration:**
```yaml
channel_enabled: "signal"
signal_phone_number: "+1234567890"
signal_cli_path: "signal-cli"
signal_config_dir: ""
signal_allowed_senders: ""
signal_rpc_port: 7583
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `signal_phone_number` | `str` | `""` | **Required.** Signal phone number (E.164 format) |
| `signal_cli_path` | `str` | `"signal-cli"` | Path to signal-cli binary |
| `signal_config_dir` | `str` | `""` | signal-cli config directory (optional) |
| `signal_allowed_senders` | `str` | `""` | Comma-separated phone numbers |
| `signal_rpc_port` | `int` | `7583` | JSON RPC socket port |
**Env vars:** `EVOSCIENTIST_SIGNAL_PHONE_NUMBER`, `EVOSCIENTIST_SIGNAL_CLI_PATH`, `EVOSCIENTIST_SIGNAL_RPC_PORT`
**Technical details:** JSON RPC over TCP socket to signal-cli daemon. Auto-starts daemon if not running (`signal-cli -u +NUMBER daemon --socket localhost:PORT`). Listens for `receive` notifications. Sends via `send` RPC method. Group detection via `groupInfo`. Mention detection via UUID matching. No public IP needed. Text chunk limit: 4096 chars.
---
### Email
**Install:** `pip install evoscientist[email]` (core dependencies included, no extras needed)
**Prerequisites:**
1. Prepare an email account with IMAP + SMTP support (Gmail, Outlook, self-hosted, etc.).
2. **Gmail:** Enable 2FA -> generate an App Password. IMAP: `imap.gmail.com:993` (SSL), SMTP: `smtp.gmail.com:587` (STARTTLS).
3. **Outlook/Office 365:** IMAP: `outlook.office365.com:993` (SSL), SMTP: `smtp.office365.com:587` (STARTTLS).
4. Ensure IMAP access is enabled in your email settings.
**Configuration:**
```yaml
channel_enabled: "email"
email_imap_host: "imap.gmail.com"
email_imap_port: 993
email_imap_username: "bot@gmail.com"
email_imap_password: "xxxx-xxxx-xxxx-xxxx"
email_imap_mailbox: "INBOX"
email_imap_use_ssl: true
email_smtp_host: "smtp.gmail.com"
email_smtp_port: 587
email_smtp_username: "bot@gmail.com"
email_smtp_password: "xxxx-xxxx-xxxx-xxxx"
email_smtp_use_tls: true
email_from_address: "bot@gmail.com"
email_poll_interval: 30
email_mark_seen: true
email_allowed_senders: ""
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `email_imap_host` | `str` | `""` | **Required.** IMAP server address |
| `email_imap_port` | `int` | `993` | IMAP port |
| `email_imap_username` | `str` | `""` | **Required.** IMAP login username |
| `email_imap_password` | `str` | `""` | **Required.** IMAP login password (or app password) |
| `email_imap_mailbox` | `str` | `"INBOX"` | Mailbox folder to monitor |
| `email_imap_use_ssl` | `bool` | `true` | Use SSL for IMAP connection |
| `email_smtp_host` | `str` | `""` | **Required.** SMTP server address |
| `email_smtp_port` | `int` | `587` | SMTP port |
| `email_smtp_username` | `str` | `""` | **Required.** SMTP login username |
| `email_smtp_password` | `str` | `""` | **Required.** SMTP login password |
| `email_smtp_use_tls` | `bool` | `true` | Use STARTTLS (`true`) or SSL (`false`) |
| `email_from_address` | `str` | `""` | Sender address (defaults to smtp_username) |
| `email_poll_interval` | `int` | `30` | IMAP poll interval in seconds |
| `email_mark_seen` | `bool` | `true` | Mark emails as read after processing |
| `email_max_body_chars` | `int` | `12000` | Max email body chars (truncated beyond) |
| `email_subject_prefix` | `str` | `"Re: "` | Reply subject prefix |
| `email_allowed_senders` | `str` | `""` | Comma-separated sender email addresses |
**Env vars:** `EVOSCIENTIST_EMAIL_IMAP_HOST`, `EVOSCIENTIST_EMAIL_IMAP_USERNAME`, `EVOSCIENTIST_EMAIL_IMAP_PASSWORD`, `EVOSCIENTIST_EMAIL_SMTP_HOST`, `EVOSCIENTIST_EMAIL_SMTP_USERNAME`, `EVOSCIENTIST_EMAIL_SMTP_PASSWORD`
**Technical details:** IMAP polling mode, checks for UNSEEN emails periodically (max 20 per cycle). Supports SSL and STARTTLS. Auto-parses multipart emails (prefers text/plain, falls back text/html -> plain text). Attachments auto-downloaded. Replies set `In-Reply-To` and `References` headers to maintain email threads. Sends HTML + plain text dual format (multipart/alternative), falls back to plain text on HTML failure. IMAP auto-reconnects on disconnect. Auth errors (auth/login/credential) not retried. No public IP needed. Text chunk limit: no limit.
---
### iMessage
**Install:** No extra Python dependencies. Requires the [imsg](https://github.com/anthropics/imsg) CLI tool.
**Requirements:** macOS only (iMessage is Apple-proprietary). Requires a signed-in Apple ID with iMessage and Full Disk Access permission for the terminal app.
**Prerequisites:**
1. Install imsg CLI:
```bash
brew install imsg
```
2. Verify: `imsg --version`
3. Ensure Messages.app is signed in and working on macOS.
**Configuration:**
```yaml
channel_enabled: "imessage"
imessage_cli_path: "imsg"
imessage_db_path: ""
imessage_service: "auto"
imessage_region: "US"
imessage_allowed_senders: ""
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `imessage_cli_path` | `str` | `"imsg"` | Path to imsg CLI binary |
| `imessage_db_path` | `str` | `""` | iMessage database path (empty = default) |
| `imessage_service` | `str` | `"auto"` | Send service: `imessage`, `sms`, or `auto` |
| `imessage_region` | `str` | `"US"` | Phone number region code |
| `imessage_allowed_senders` | `str` | `""` | Comma-separated allowlist (see below) |
**Allowlist formats:** phone (`+1234567890`), email (`user@example.com`), `chat_id:123`, `chat_guid:iMessage;-;+1234567890`, wildcard `*`.
**Env vars:** `EVOSCIENTIST_IMESSAGE_CLI_PATH`, `EVOSCIENTIST_IMESSAGE_SERVICE`, `EVOSCIENTIST_IMESSAGE_ALLOWED_SENDERS`
**Technical details:** JSON-RPC over stdio with imsg CLI. Creates `watch.subscribe` on startup for real-time message streaming (not polling). Supports iMessage + SMS dual channel (`service: auto`). Target resolution supports chat_id, chat_guid, chat_identifier, and phone/email. Attachments read from local paths provided by imsg. Group detection via `is_group` field. RPC errors (AppleScript/permission/not found) not retried; only connection timeouts retried. Plain text format (no Markdown). No public IP needed. Text chunk limit: 4000 chars.
---
## Running Multiple Channels
Comma-separate channel names in the config to enable multiple channels simultaneously:
```yaml
channel_enabled: "telegram,discord,slack"
```
All enabled channels run concurrently via the internal `MessageBus`. Each channel:
- Has its own connection lifecycle (connect, reconnect, health check)
- Shares the same `InboundConsumer` worker pool and agent instance
- Routes outbound replies back to the originating channel automatically
### Multi-Channel Architecture
```
ChannelManager
├── TelegramChannel ──┐
├── DiscordChannel ──┤
├── SlackChannel ──┤──► MessageBus ──► InboundConsumer ──► Agent
├── FeishuChannel ──┤ │
└── ... ──┘ ▼
OutboundMessage
│
Dispatcher routes
to origin channel
```
### Health Monitoring
The `ChannelManager` runs a background health check task that monitors all active channels. Access health status via:
```bash
# CLI
EvoSci channel status
# HTTP (if health endpoint is enabled)
curl http://localhost:8080/healthz
```
## Docker Deployment
For webhook-based channels (Feishu, WeChat), Docker simplifies port mapping and process management:
```dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY . .
RUN pip install evoscientist[feishu,wechat]
# Expose webhook ports
EXPOSE 9000 9001
CMD ["EvoSci", "serve"]
```
```bash
docker build -t evoscientist .
docker run -d \
-p 9000:9000 \
-p 9001:9001 \
-e EVOSCIENTIST_CHANNEL_ENABLED="feishu,wechat" \
-e EVOSCIENTIST_FEISHU_APP_ID="cli_xxx" \
-e EVOSCIENTIST_FEISHU_APP_SECRET="xxx" \
-e EVOSCIENTIST_WECHAT_BACKEND="wecom" \
-e EVOSCIENTIST_WECHAT_WECOM_CORP_ID="ww..." \
-e EVOSCIENTIST_WECHAT_WECOM_SECRET="xxx" \
evoscientist
```
For polling/WebSocket channels (Telegram, Discord, Slack, DingTalk, QQ), no port mapping is needed:
```bash
docker run -d \
-e EVOSCIENTIST_CHANNEL_ENABLED="telegram" \
-e EVOSCIENTIST_TELEGRAM_BOT_TOKEN="123456:ABC-xxx" \
evoscientist
```
## Troubleshooting
### Common Issues
**Bot not responding to messages**
1. Check that `channel_enabled` includes your channel name
2. Verify the bot token/credentials are correct: `EvoSci config get telegram_bot_token`
3. If using `allowed_senders`, ensure your user ID is listed
4. For group chats, ensure the bot is @mentioned (default behavior)
5. Check logs for middleware drops: `DedupMiddleware`, `AllowListMiddleware`, or `MentionGatingMiddleware`
**"channel X not found" or import errors**
Install the channel-specific dependencies:
```bash
pip install evoscientist[telegram] # or discord, slack, feishu, etc.
```
**Webhook channels (Feishu, WeChat) not receiving messages**
1. Ensure the webhook URL is publicly reachable (not behind NAT without port forwarding)
2. For local development, use a tunnel: `ngrok http 9000` or [natapp](https://natapp.cn/) for China
3. Verify the callback URL matches exactly (including path: `/webhook/event` for Feishu, `/wechat/callback` for WeChat)
4. Check that signature verification tokens match between the platform config and your local config
**WeChat API error `60020`**
Add your server's public IP to the WeCom app's **Trusted IP** list in the admin console.
**Token refresh failures**
- Feishu/WeChat/DingTalk tokens auto-refresh with a 5-minute safety margin before expiry
- If the refresh endpoint is unreachable (network issues), messages will fail until the next successful refresh
- Check proxy settings if your server requires a proxy to reach external APIs
**Duplicate responses**
- The `DedupMiddleware` prevents most duplicates using a 60-second LRU cache
- If you see duplicates, check if the platform is sending the same message with different IDs (some platforms retry delivery)
**Messages truncated**
- Each platform has a max text length (see Capability Matrix)
- Long responses are automatically chunked at paragraph/code-block boundaries
- Adjust `text_chunk_limit` in the channel config if needed
### Debug Logging
Enable debug logs for the channel subsystem:
```bash
export EVOSCIENTIST_LOG_LEVEL=DEBUG
# or
EvoSci config set log_level debug
```
Channel-specific log output is prefixed with the module path (e.g., `EvoScientist.channels.telegram.channel`).
+49
View File
@@ -0,0 +1,49 @@
"""Communication channels for EvoScientist.
This module provides an extensible interface for different messaging channels
(iMessage, Telegram, Discord, Slack, WeChat, DingTalk, Feishu, Email, QQ, Signal) to communicate with the EvoScientist agent.
"""
from .base import Channel, IncomingMessage, OutgoingMessage, RawIncoming, chunk_text
from .bus import InboundMessage, MessageBus, OutboundMessage
from .capabilities import ChannelCapabilities
from .channel_manager import (
ChannelManager,
available_channels,
create_channel,
register_channel,
)
from .consumer import InboundConsumer
from .formatter import UnifiedFormatter
from .middleware import TypingManager
from .plugin import ChannelMeta, ChannelPlugin, ReloadPolicy
from .standalone import run_standalone
# Backward compat: ChannelServer is now Channel itself
ChannelServer = Channel
__all__ = [
"Channel",
# New modules
"ChannelCapabilities",
"ChannelManager",
"ChannelMeta",
# Plugin architecture
"ChannelPlugin",
"ChannelServer",
"InboundConsumer",
"InboundMessage",
"IncomingMessage",
"MessageBus",
"OutboundMessage",
"OutgoingMessage",
"RawIncoming",
"ReloadPolicy",
"TypingManager",
"UnifiedFormatter",
"available_channels",
"chunk_text",
"create_channel",
"register_channel",
"run_standalone",
]
File diff suppressed because it is too large Load Diff
+6
View File
@@ -0,0 +1,6 @@
"""Message bus for decoupled channel-agent communication."""
from .events import InboundMessage, OutboundMessage
from .message_bus import MessageBus
__all__ = ["InboundMessage", "MessageBus", "OutboundMessage"]
+52
View File
@@ -0,0 +1,52 @@
"""Event types for the message bus."""
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any
@dataclass
class InboundMessage:
"""Message received from a chat channel.
Carries enough context for the bus to route and for the agent
to build a session: which channel, who sent it, which chat.
"""
channel: str
sender_id: str
chat_id: str
content: str
timestamp: datetime = field(default_factory=datetime.now)
message_id: str = ""
media: list[str] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
is_group: bool = False
was_mentioned: bool = True
@property
def sender(self) -> str:
"""Alias for ``sender_id`` (compatibility with IncomingMessage)."""
return self.sender_id
@property
def session_key(self) -> str:
"""Unique key for session identification: ``channel:chat_id``."""
return f"{self.channel}:{self.chat_id}"
@dataclass
class OutboundMessage:
"""Message to send to a chat channel."""
channel: str
chat_id: str
content: str
reply_to: str | None = None
media: list[str] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
@property
def recipient(self) -> str:
"""Alias for ``chat_id`` (compatibility with OutgoingMessage)."""
return self.chat_id
+114
View File
@@ -0,0 +1,114 @@
"""Async message bus that decouples chat channels from the agent core.
Channels push messages to the inbound queue; the agent (or any consumer)
reads from inbound, processes, and pushes responses to the outbound queue.
A background dispatcher routes outbound messages to the correct channel
via subscriber callbacks.
Deduplication is handled at the Channel level (single dedup point).
"""
import asyncio
import logging
from collections.abc import Awaitable, Callable
from ..debug import TraceMixin, debug_trace_enabled
from .events import InboundMessage, OutboundMessage
logger = logging.getLogger(__name__)
OutboundCallback = Callable[[OutboundMessage], Awaitable[None]]
class MessageBus(TraceMixin):
"""Async message bus that decouples chat channels from the agent core."""
name = "bus"
def __init__(self):
self.inbound: asyncio.Queue[InboundMessage] = asyncio.Queue(maxsize=5000)
self.outbound: asyncio.Queue[OutboundMessage] = asyncio.Queue(maxsize=5000)
self._outbound_subscribers: dict[str, list[OutboundCallback]] = {}
self._running = False
self._debug_trace = debug_trace_enabled()
self._trace_logger = logger
# ── inbound (channel → agent) ──
async def publish_inbound(self, msg: InboundMessage) -> None:
"""Publish a message from a channel to the agent."""
await self.inbound.put(msg)
async def consume_inbound(self) -> InboundMessage:
"""Consume the next inbound message (blocks until available)."""
return await self.inbound.get()
# ── outbound (agent → channel) ──
async def publish_outbound(self, msg: OutboundMessage) -> None:
"""Publish a response from the agent to channels."""
await self.outbound.put(msg)
async def consume_outbound(self) -> OutboundMessage:
"""Consume the next outbound message (blocks until available)."""
return await self.outbound.get()
# ── subscriber routing ──
def subscribe_outbound(
self,
channel: str,
callback: OutboundCallback,
) -> None:
"""Register a callback for outbound messages targeting *channel*."""
if channel not in self._outbound_subscribers:
self._outbound_subscribers[channel] = []
self._outbound_subscribers[channel].append(callback)
async def dispatch_outbound(self) -> None:
"""Route outbound messages to subscribed channels.
Run as a background task — loops until :meth:`stop` is called.
"""
self._running = True
while self._running:
try:
msg = await asyncio.wait_for(
self.outbound.get(),
timeout=1.0,
)
except TimeoutError:
continue
subscribers = self._outbound_subscribers.get(msg.channel, [])
if not subscribers:
self._trace_event(
"bus_dispatch_drop",
target_channel=msg.channel,
reason="no_subscriber",
chat_id=msg.chat_id,
)
logger.warning(f"No subscriber for channel: {msg.channel}")
continue
for callback in subscribers:
try:
await callback(msg)
except Exception as e:
self._trace_event(
"bus_dispatch_error",
target_channel=msg.channel,
chat_id=msg.chat_id,
error_type=type(e).__name__,
)
logger.error(f"Error dispatching to {msg.channel}: {e}")
def stop(self) -> None:
"""Stop the dispatcher loop."""
self._running = False
@property
def inbound_size(self) -> int:
return self.inbound.qsize()
@property
def outbound_size(self) -> int:
return self.outbound.qsize()
+220
View File
@@ -0,0 +1,220 @@
"""Channel capabilities declaration system.
Each channel declares its capabilities via a ChannelCapabilities dataclass,
enabling the framework to adapt behavior automatically (formatting, reactions,
streaming, threading, etc.) without per-channel branching in core logic.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
FormatType = Literal["html", "markdown", "slack_mrkdwn", "discord", "plain"]
@dataclass(frozen=True)
class ChannelCapabilities:
"""Immutable declaration of what a channel supports.
Set once as a class attribute on each Channel subclass.
The framework inspects these at runtime to auto-configure behavior.
"""
# ── Messaging features ──────────────────────────────────────────
format_type: FormatType = "plain"
max_text_length: int = 4096
max_file_size: int = 20 * 1024 * 1024 # 20 MB
# ── Interaction capabilities ────────────────────────────────────
streaming: bool = False # edit-in-place streaming output
threading: bool = False # message threads / topics
reactions: bool = False # emoji reactions on messages
typing: bool = False # typing indicator API
inline_buttons: bool = False # inline keyboard / action buttons
# ── Media capabilities ──────────────────────────────────────────
media_send: bool = False # can send files/images
media_receive: bool = False # can receive files/images
voice: bool = False # platform has voice/audio messages that arrive as downloadable files (receive only, not bot sending)
stickers: bool = False # supports sticker receive (not bot sending)
location: bool = False # supports location message receive (not bot sending)
video: bool = False # video messages
# ── Group features ──────────────────────────────────────────────
groups: bool = False # group chat support
mentions: bool = False # @mention detection
# ── Rich text ───────────────────────────────────────────────────
markdown: bool = False # supports Markdown rendering
html: bool = False # supports HTML rendering
# ── Extended capabilities ────────────────────────────────────────
chat_types: tuple[str, ...] = () # ("direct", "group", "channel", "thread")
edit: bool = False # message editing after send
unsend: bool = False # message recall / unsend
block_streaming: bool = False # block edit-in-place streaming
native_commands: bool = False # platform-native slash commands
polls: bool = False # poll / vote messages
def supports(self, feature: str) -> bool:
"""Check if a feature is supported by name."""
return getattr(self, feature, False)
# ═════════════════════════════════════════════════════════════════════
# Pre-built capability profiles for each channel
# ═════════════════════════════════════════════════════════════════════
TELEGRAM = ChannelCapabilities(
format_type="html",
max_text_length=4000,
streaming=False, # could edit messages, but not implemented yet
threading=False, # topics exist but not used yet
reactions=True,
typing=True,
media_send=True,
media_receive=True,
voice=True,
stickers=True,
location=True,
groups=True,
mentions=True,
html=True,
chat_types=("direct", "group", "channel"),
edit=True,
unsend=True,
native_commands=True,
polls=True,
)
DISCORD = ChannelCapabilities(
format_type="discord",
max_text_length=2000,
streaming=False,
threading=True,
reactions=True,
typing=True,
media_send=True,
media_receive=True,
voice=False, # no distinct voice message type in Discord bot API
groups=True,
mentions=True,
markdown=True,
chat_types=("direct", "group", "thread"),
edit=True,
unsend=True,
native_commands=True,
polls=True,
)
SLACK = ChannelCapabilities(
format_type="slack_mrkdwn",
max_text_length=4000,
streaming=False,
threading=True,
reactions=True,
typing=False, # no native typing API; workaround via post+delete "..." message
media_send=True,
media_receive=True,
voice=False, # no distinct voice message type in Slack bot API
groups=True,
mentions=True,
chat_types=("direct", "group", "thread"),
edit=True,
unsend=True,
native_commands=True,
)
FEISHU = ChannelCapabilities(
format_type="markdown",
max_text_length=4096,
reactions=True,
typing=False, # no typing API
media_send=True,
media_receive=True,
voice=True,
stickers=True,
groups=True,
mentions=True,
markdown=True,
chat_types=("direct", "group"),
edit=True,
unsend=True,
)
DINGTALK = ChannelCapabilities(
format_type="markdown",
max_text_length=4096,
typing=False, # no typing API for bots
media_send=True,
media_receive=True,
voice=True,
groups=True,
mentions=True,
markdown=True,
chat_types=("direct", "group"),
)
QQ = ChannelCapabilities(
format_type="plain",
max_text_length=4096,
typing=False, # no typing API for QQ bots
media_send=True,
media_receive=True,
voice=False, # qq-botpy does not expose voice as a distinct message type
groups=True,
mentions=True,
chat_types=("direct", "group", "channel"),
unsend=True,
)
WECHAT = ChannelCapabilities(
format_type="markdown", # WeCom supports markdown
max_text_length=4096,
typing=False, # no typing API
media_send=True,
media_receive=True,
voice=True,
location=True,
groups=True,
mentions=True,
markdown=True,
chat_types=("direct", "group"),
unsend=True,
)
SIGNAL = ChannelCapabilities(
format_type="plain",
max_text_length=4096,
reactions=True,
typing=True,
media_send=True,
media_receive=True,
voice=True,
groups=True,
mentions=True,
chat_types=("direct", "group"),
)
EMAIL = ChannelCapabilities(
format_type="html",
max_text_length=999_999, # no practical limit
media_send=True,
media_receive=True,
html=True,
chat_types=("direct",),
)
IMESSAGE = ChannelCapabilities(
format_type="plain",
max_text_length=999_999,
typing=False, # Apple does not expose typing indicator API
media_send=True,
media_receive=True,
voice=True,
groups=True,
mentions=False, # iMessage has no @mention concept
reactions=False, # imsg CLI cannot send tapback reactions
chat_types=("direct", "group"),
)
File diff suppressed because it is too large Load Diff
+128
View File
@@ -0,0 +1,128 @@
"""Base configuration for all channel implementations.
Provides common fields shared across channels, reducing duplication.
Channel-specific configs inherit from BaseChannelConfig.
Also provides ready-made ConfigAdapter implementations for the two most
common account patterns:
- ``SingleAccountConfigAdapter`` — one account per channel (default).
- ``MultiAccountConfigAdapter`` — multiple accounts from a config dict.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
@dataclass
class BaseChannelConfig:
"""Common configuration fields for all channels.
Subclass this for channel-specific configs. Only add fields
here that are used by 3+ channels.
"""
allowed_senders: set[str] | None = None
allowed_channels: set[str] | None = None
text_chunk_limit: int = 4096
proxy: str | None = None
include_attachments: bool = True
debug_trace: bool = False
accounts: dict | None = None # multi-account config mapping
class SingleAccountConfigAdapter:
"""For channels that only ever have one account (most channels).
Returns a single ``"default"`` account whose config is the entire
channel config object. This is the zero-change default: existing
single-account channels get multi-account support for free.
"""
def list_account_ids(self, config: Any) -> list[str]:
return ["default"]
def resolve_account(
self,
config: Any,
account_id: str | None = None,
) -> Any:
return config
def is_enabled(self, account: Any, config: Any) -> bool:
return True
def is_configured(self, account: Any, config: Any) -> bool:
"""Check that the account has at least some non-None values."""
if account is None:
return False
if isinstance(account, dict):
return bool(account)
# dataclass / object — check that at least one field is truthy
if hasattr(account, "__dataclass_fields__"):
return any(getattr(account, f, None) for f in account.__dataclass_fields__)
return True
class MultiAccountConfigAdapter:
"""For channels that support multiple accounts.
Expects the channel config to contain a mapping of accounts under
a configurable key (default ``"accounts"``). Each entry is keyed
by account id and holds account-specific settings.
Example config structure::
{
"accounts": {
"bot1": {"token": "...", "enabled": true},
"bot2": {"token": "...", "enabled": false},
}
}
"""
def __init__(
self,
accounts_key: str = "accounts",
required_fields: list[str] | None = None,
) -> None:
self._accounts_key = accounts_key
self._required_fields = required_fields or []
def _get_accounts_map(self, config: Any) -> dict[str, Any]:
"""Extract the accounts mapping from config."""
if isinstance(config, dict):
return config.get(self._accounts_key, {})
return getattr(config, self._accounts_key, None) or {}
def list_account_ids(self, config: Any) -> list[str]:
return list(self._get_accounts_map(config).keys())
def resolve_account(
self,
config: Any,
account_id: str | None = None,
) -> Any:
accounts = self._get_accounts_map(config)
if account_id is None:
# Return the first account, or empty dict
return next(iter(accounts.values()), {})
return accounts.get(account_id, {})
def is_enabled(self, account: Any, config: Any) -> bool:
if isinstance(account, dict):
return account.get("enabled", True)
return getattr(account, "enabled", True)
def is_configured(self, account: Any, config: Any) -> bool:
if not account:
return False
for f in self._required_fields:
if isinstance(account, dict):
if not account.get(f):
return False
elif not getattr(account, f, None):
return False
return True
+888
View File
@@ -0,0 +1,888 @@
"""Unified inbound message consumer.
Provides :class:`InboundConsumer` — a single class that consumes
inbound messages from the :class:`MessageBus`, runs them through
the agent, and publishes outbound responses. This replaces the
inline consumer loops that were duplicated in ``cli.py`` and
``standalone.py``.
"""
from __future__ import annotations
import asyncio
import logging
import time
import uuid
from collections import OrderedDict
from collections.abc import AsyncIterator, Callable
from dataclasses import dataclass
from typing import Any, TypeVar
from .base import Channel
from .bus import MessageBus
from .bus.events import InboundMessage, OutboundMessage
logger = logging.getLogger(__name__)
T = TypeVar("T")
_MAX_CHAT_LOCKS = 10_000
_MAX_SESSIONS = 10_000
_MAX_HITL_ROUNDS = 50
_HITL_APPROVAL_TIMEOUT = 120.0 # seconds to wait for HITL approval reply
_ASK_USER_TIMEOUT = (
300.0 # seconds to wait for ask_user reply (longer for thinking time)
)
@dataclass
class ConsumerMetrics:
"""Cumulative processing counters for the consumer."""
total_processed: int = 0
total_successes: int = 0
total_failures: int = 0
total_timeouts: int = 0
async def _timeout_aiter(
agen: AsyncIterator[T],
idle_timeout: float,
) -> AsyncIterator[T]:
"""Wrap an async iterator with a per-yield idle timeout.
If ``__anext__()`` does not produce a value within *idle_timeout*
seconds, :class:`asyncio.TimeoutError` is raised. Continuous
yielding resets the timer each time, so only a truly stalled
generator will trigger the timeout.
"""
ait = agen.__aiter__()
try:
while True:
try:
item = await asyncio.wait_for(ait.__anext__(), timeout=idle_timeout)
except StopAsyncIteration:
return
yield item
finally:
if hasattr(ait, "aclose"):
await ait.aclose()
def _format_todo_list(todos: list[dict]) -> str:
"""Format todo items as a numbered list."""
lines = ["\U0001f4cb Todo List\n"] # 📋
for i, item in enumerate(todos, 1):
content = item.get("content", "")
lines.append(f"{i}. {content}")
lines.append(f"\n\U0001f680 {len(todos)} tasks") # 🚀
return "\n".join(lines)
def _join_subagent_text(buffers: dict[str, tuple[str, list[str]]]) -> str:
"""Join sub-agent text buffers into a single fallback string.
*buffers* maps ``instance_id`` → ``(display_name, chunks)``.
When only one instance produced text, return its content directly.
When multiple instances share the same display name, number them
(e.g. ``[research-agent #1]``, ``[research-agent #2]``).
"""
if not buffers:
return ""
if len(buffers) == 1:
_display_name, chunks = next(iter(buffers.values()))
return "".join(chunks)
# Group by display_name to detect same-name instances
name_groups: dict[str, list[list[str]]] = {}
for _instance_id, (display_name, chunks) in buffers.items():
name_groups.setdefault(display_name, []).append(chunks)
sections: list[str] = []
for display_name, chunk_lists in name_groups.items():
if len(chunk_lists) == 1:
sections.append(f"[{display_name}]: {''.join(chunk_lists[0])}")
else:
for i, chs in enumerate(chunk_lists, 1):
sections.append(f"[{display_name} #{i}]: {''.join(chs)}")
return "\n\n".join(sections)
def _should_auto_approve(action_requests: list[dict]) -> bool:
"""Check if all action requests can be auto-approved via config.
Returns True if no manual approval is needed (config auto_approve,
non-execute tools, or shell_allow_list match).
"""
if not action_requests:
return True
try:
from ..config.settings import load_config
cfg = load_config()
except Exception:
return False # fail-closed
if cfg.auto_approve:
return True
shell_allow_list = (
[s.strip() for s in cfg.shell_allow_list.split(",") if s.strip()]
if cfg.shell_allow_list
else []
)
for req in action_requests:
name = (
req.get("name", "") if isinstance(req, dict) else getattr(req, "name", "")
)
if name != "execute":
continue
args = (
req.get("args", {}) if isinstance(req, dict) else getattr(req, "args", {})
)
command = args.get("command", "") if isinstance(args, dict) else ""
cmd = command.strip()
if not any(cmd.startswith(prefix) for prefix in shell_allow_list):
return False
return True
def _format_approval_prompt(action_requests: list[dict]) -> str:
"""Format an approval prompt as a text message for channel users."""
lines = ["\u26a0\ufe0f Approval Required\n"]
for i, req in enumerate(action_requests, 1):
name = (
req.get("name", "") if isinstance(req, dict) else getattr(req, "name", "")
)
args = (
req.get("args", {}) if isinstance(req, dict) else getattr(req, "args", {})
)
if isinstance(args, dict):
command = args.get("command", args.get("path", ""))
else:
command = ""
if command:
lines.append(f" {i}. {name}: {command}")
else:
lines.append(f" {i}. {name}")
lines.append("")
lines.append("Reply: 1=Approve, 2=Reject, 3=Approve all")
lines.append("(Auto-reject in 2 min if no reply)")
return "\n".join(lines)
def _parse_approval_reply(text: str) -> str | None:
"""Parse a channel user's reply as an approval decision.
Returns "approve", "reject", "auto", or None if not recognized.
"""
t = text.strip().lower()
if t in ("1", "y", "yes", "approve", "ok"):
return "approve"
if t in ("2", "n", "no", "reject"):
return "reject"
if t in ("3", "a", "auto", "approve all"):
return "auto"
return None
@dataclass
class _PendingInterrupt:
"""Stored state for a pending HITL interrupt awaiting channel user reply."""
thread_id: str
action_requests: list
event: asyncio.Event # set when user replies
decision: str | None = None # "approve", "reject", "auto"
@dataclass
class _PendingAskUserReply:
"""Stored state for a pending ask_user question awaiting channel user reply."""
event: asyncio.Event # set when user replies
reply: str | None = None # raw reply text
class InboundConsumer:
"""Consume inbound messages from the bus, process via agent, publish outbound.
Parameters
----------
bus:
The MessageBus to consume from / publish to.
manager:
The ChannelManager (used to look up channel instances).
agent:
The agent object (must support ``stream_agent_events``).
thread_id:
Default thread ID for agent conversations.
send_thinking:
Whether to forward thinking messages to the channel.
on_message_received:
Optional callback ``(msg: InboundMessage) -> None`` invoked when
a message is consumed (e.g. for CLI Rich display).
on_streaming_event:
Optional callback ``(event: dict) -> None`` invoked for each
streaming event from the agent.
on_message_sent:
Optional callback ``(msg: OutboundMessage) -> None`` invoked when
the outbound message is published.
inference_timeout:
Per-yield idle timeout in seconds for the agent stream. If the
agent produces no event for this long, the inference is aborted.
max_concurrent:
Number of worker coroutines (= max parallel inferences).
max_pending:
Maximum depth of the internal work queue. When full, the
consumer loop blocks (back-pressure).
drain_timeout:
Seconds to wait for in-flight workers to finish during ``stop()``.
"""
def __init__(
self,
bus: MessageBus,
manager: Any,
agent: Any,
thread_id: str,
*,
send_thinking: bool = False,
on_message_received: Callable[[InboundMessage], None] | None = None,
on_streaming_event: Callable[[dict], None] | None = None,
on_message_sent: Callable[[OutboundMessage], None] | None = None,
inference_timeout: float = 300.0,
max_concurrent: int = 5,
max_pending: int = 50,
drain_timeout: float = 30.0,
):
self.bus = bus
self.manager = manager
self.agent = agent
self.thread_id = thread_id
self.send_thinking = send_thinking
self._on_message_received = on_message_received
self._on_streaming_event = on_streaming_event
self._on_message_sent = on_message_sent
self._sessions: OrderedDict[str, str] = (
OrderedDict()
) # sender_id -> thread_id (LRU)
# Per-chat locks: same chat is processed serially (bounded)
self._chat_locks: dict[str, asyncio.Lock] = {}
# Inference timeout
self._inference_timeout = inference_timeout
# Worker pool
self._max_concurrent = max_concurrent
self._work_queue: asyncio.Queue[InboundMessage | None] = asyncio.Queue(
maxsize=max_pending,
)
self._workers: list[asyncio.Task] = []
self._stopping = False
self._drain_timeout = drain_timeout
# Metrics
self._metrics = ConsumerMetrics()
# HITL: pending interrupts per session_key, and auto-approve sessions
self._pending_interrupts: dict[str, _PendingInterrupt] = {}
self._auto_approve_sessions: set[str] = set()
# ask_user: pending reply per session_key
self._pending_ask_user_replies: dict[str, _PendingAskUserReply] = {}
def _get_thread_id(self, sender_id: str) -> str:
"""Get or create a thread ID for the given sender.
Uses LRU ordering: recently accessed senders are moved to the
end, so eviction always removes the least-recently-active sender.
"""
if sender_id in self._sessions:
self._sessions.move_to_end(sender_id)
return self._sessions[sender_id]
if len(self._sessions) >= _MAX_SESSIONS:
# Evict the least-recently-used entry
self._sessions.popitem(last=False)
if self.thread_id:
self._sessions[sender_id] = f"{self.thread_id}:{sender_id}"
else:
self._sessions[sender_id] = str(uuid.uuid4())
return self._sessions[sender_id]
def _get_channel(self, channel_name: str) -> Channel | None:
"""Look up the channel by name from the manager."""
return self.manager.get_channel(channel_name)
# ── lifecycle ──
async def run(self) -> None:
"""Main consumer loop — runs until ``stop()`` or cancellation.
Spawns *max_concurrent* worker coroutines that pull from an
internal bounded queue. The loop reads from the bus and feeds
the queue; when the queue is full the loop blocks (back-pressure).
"""
self._stopping = False
self._workers = [
asyncio.create_task(self._worker(i)) for i in range(self._max_concurrent)
]
try:
while not self._stopping:
try:
msg = await asyncio.wait_for(
self.bus.consume_inbound(),
timeout=1.0,
)
except TimeoutError:
continue
except asyncio.CancelledError:
break
if self._stopping:
break
await self._work_queue.put(msg) # blocks when full (back-pressure)
finally:
if not self._stopping:
await self.stop()
async def stop(self) -> None:
"""Gracefully drain in-flight work and shut down workers."""
self._stopping = True
logger.info("Consumer stopping: draining in-flight messages...")
pending_count = self._work_queue.qsize()
# Send a None sentinel per worker so each exits its loop
for _ in self._workers:
try:
self._work_queue.put_nowait(None)
except asyncio.QueueFull:
pass
# Wait for workers to finish, then force-cancel stragglers
if self._workers:
done, still_running = await asyncio.wait(
self._workers,
timeout=self._drain_timeout,
)
for task in still_running:
task.cancel()
try:
await task
except (asyncio.CancelledError, Exception):
pass
logger.info(
f"Consumer drain: {len(done)} finished, "
f"{len(still_running)} force-cancelled, "
f"{pending_count} were pending"
)
self._workers.clear()
# ── workers ──
async def _worker(self, worker_id: int) -> None:
"""Pull messages from the work queue and process them."""
while True:
msg = await self._work_queue.get()
if msg is None:
break # shutdown sentinel
try:
await self._handle_message(msg)
except Exception:
logger.exception(f"Worker {worker_id} unhandled error")
finally:
self._work_queue.task_done()
async def _handle_message(self, msg: InboundMessage) -> None:
"""Process a single inbound message."""
if self._on_message_received:
try:
self._on_message_received(msg)
except Exception:
pass
channel = self._get_channel(msg.channel)
thread_id = self._get_thread_id(msg.sender_id)
session_key = msg.session_key # "channel:chat_id"
# Lazily create per-chat lock; evict stale locks when too many
if session_key not in self._chat_locks:
self._chat_locks[session_key] = asyncio.Lock()
if len(self._chat_locks) > _MAX_CHAT_LOCKS:
self._evict_chat_locks()
self._metrics.total_processed += 1
# ask_user: check if this message is a reply to a pending question.
# Must be checked BEFORE HITL approval — any text is a valid answer.
if session_key in self._pending_ask_user_replies:
pending_ask = self._pending_ask_user_replies[session_key]
pending_ask.reply = msg.content
pending_ask.event.set()
return # consumed as ask_user answer
# HITL: check if this message is a reply to a pending approval
if session_key in self._pending_interrupts:
pending = self._pending_interrupts[session_key]
decision = _parse_approval_reply(msg.content)
if decision is not None:
pending.decision = decision
pending.event.set()
return # don't process as a new agent message
# Unrecognized reply — treat as new message, cancel pending
pending.decision = "reject"
pending.event.set()
del self._pending_interrupts[session_key]
async with self._chat_locks[session_key]:
await self._stream_with_hitl(msg, channel, thread_id, session_key)
async def _stream_with_hitl(
self,
msg: InboundMessage,
channel: Channel | None,
thread_id: str,
session_key: str,
) -> None:
"""Stream agent events with HITL interrupt handling."""
from ..stream.events import stream_agent_events
stream_input: Any = msg.content
_t0 = time.monotonic()
try:
if channel:
await channel.start_typing(msg.chat_id)
for _hitl_round in range(_MAX_HITL_ROUNDS):
final_content = ""
thinking_buffer: list[str] = []
todo_sent = False
subagent_text_buffers: dict[str, tuple[str, list[str]]] = {}
thinking_sent = False
interrupt_data: dict | None = None
async for event in _timeout_aiter(
stream_agent_events(
self.agent,
stream_input,
thread_id,
media=msg.media or None
if isinstance(stream_input, str)
else None,
),
self._inference_timeout,
):
event_type = event.get("type")
if self._on_streaming_event:
try:
self._on_streaming_event(event)
except Exception:
pass
if event_type == "thinking":
thinking_text = event.get("content", "")
if thinking_text:
thinking_buffer.append(thinking_text)
elif event_type == "tool_call":
if event.get("name") == "write_todos" and not todo_sent:
todos = event.get("args", {}).get("todos", [])
if todos and channel:
if thinking_buffer and not thinking_sent:
full_thinking = "".join(thinking_buffer)
if full_thinking:
await channel.send_thinking_message(
msg.sender_id,
full_thinking,
msg.metadata,
)
thinking_sent = True
thinking_buffer.clear()
await channel.send_todo_message(
msg.sender_id,
_format_todo_list(todos),
msg.metadata,
)
todo_sent = True
elif event_type == "text":
final_content += event.get("content", "")
elif event_type == "progress":
# Internal agent planning — ignore for channel delivery.
# This text should NOT appear in the user-facing response.
pass
elif event_type == "subagent_text":
sa_name = event.get("subagent", "unknown")
instance_id = event.get("instance_id") or sa_name
if instance_id not in subagent_text_buffers:
subagent_text_buffers[instance_id] = (sa_name, [])
subagent_text_buffers[instance_id][1].append(
event.get("content", "")
)
elif event_type == "done":
final_content = event.get("content", "") or final_content
elif event_type == "interrupt":
interrupt_data = event
break # exit async for to handle interrupt
elif event_type == "ask_user":
interrupt_data = event
break # exit async for to handle ask_user
# Flush thinking
if thinking_buffer and not thinking_sent and channel:
full_thinking = "".join(thinking_buffer)
if full_thinking:
await channel.send_thinking_message(
msg.sender_id,
full_thinking,
msg.metadata,
)
# No interrupt — normal completion
if interrupt_data is None:
outbound = OutboundMessage(
channel=msg.channel,
chat_id=msg.chat_id,
content=final_content
or _join_subagent_text(subagent_text_buffers)
or "No response",
reply_to=msg.message_id or None,
metadata=msg.metadata,
)
await self.bus.publish_outbound(outbound)
self._metrics.total_successes += 1
elapsed = time.monotonic() - _t0
logger.info(
"stream completed: %d chars, %.2fs, session=%s",
len(outbound.content),
elapsed,
session_key,
)
if self._on_message_sent:
try:
self._on_message_sent(outbound)
except Exception:
pass
return # done
# ask_user: send questions to channel user, collect answers
if interrupt_data.get("type") == "ask_user":
result = await self._resolve_ask_user(
msg,
interrupt_data,
session_key,
)
from langgraph.types import Command # type: ignore[import-untyped]
stream_input = Command(resume=result)
continue
# HITL: resolve the interrupt
action_reqs = interrupt_data.get("action_requests", [])
n = len(action_reqs) or 1
# Session auto-approve (user previously chose "Approve all")
if session_key in self._auto_approve_sessions:
from langgraph.types import Command # type: ignore[import-untyped]
stream_input = Command(
resume={"decisions": [{"type": "approve"} for _ in range(n)]}
)
continue
# Config auto-approve (auto_approve, non-execute, allow_list)
if _should_auto_approve(action_reqs):
from langgraph.types import Command # type: ignore[import-untyped]
stream_input = Command(
resume={"decisions": [{"type": "approve"} for _ in range(n)]}
)
continue
# Needs user approval — send prompt to channel
prompt_text = _format_approval_prompt(action_reqs)
await self.bus.publish_outbound(
OutboundMessage(
channel=msg.channel,
chat_id=msg.chat_id,
content=prompt_text,
metadata=msg.metadata,
)
)
# Wait for user reply
pending = _PendingInterrupt(
thread_id=thread_id,
action_requests=action_reqs,
event=asyncio.Event(),
)
self._pending_interrupts[session_key] = pending
try:
await asyncio.wait_for(
pending.event.wait(),
timeout=_HITL_APPROVAL_TIMEOUT,
)
except TimeoutError:
# Auto-approve on timeout
pending.decision = "approve"
finally:
self._pending_interrupts.pop(session_key, None)
decision = pending.decision or "approve"
if decision == "reject":
await self.bus.publish_outbound(
OutboundMessage(
channel=msg.channel,
chat_id=msg.chat_id,
content="Tool execution rejected.",
metadata=msg.metadata,
)
)
return
if decision == "auto":
self._auto_approve_sessions.add(session_key)
from langgraph.types import Command # type: ignore[import-untyped]
stream_input = Command(
resume={"decisions": [{"type": "approve"} for _ in range(n)]}
)
# continue to next HITL round
except TimeoutError:
self._metrics.total_timeouts += 1
elapsed = time.monotonic() - _t0
logger.error(
"Inference timeout (%ds idle) for %s in %s, elapsed=%.2fs, response=%d chars",
self._inference_timeout,
msg.sender_id,
session_key,
elapsed,
len(final_content),
)
await self.bus.publish_outbound(
OutboundMessage(
channel=msg.channel,
chat_id=msg.chat_id,
content="Sorry, the response timed out. Please try again.",
metadata=msg.metadata,
)
)
except Exception as e:
self._metrics.total_failures += 1
elapsed = time.monotonic() - _t0
logger.error(
"Agent error: %s | elapsed=%.2fs, response=%d chars, session=%s",
e,
elapsed,
len(final_content),
session_key,
)
await self.bus.publish_outbound(
OutboundMessage(
channel=msg.channel,
chat_id=msg.chat_id,
content="Sorry, something went wrong. Please try again later.",
metadata=msg.metadata,
)
)
finally:
if channel:
await channel.stop_typing(msg.chat_id)
# ── observability ──
@property
def pending_count(self) -> int:
"""Number of messages waiting in the work queue."""
return self._work_queue.qsize()
@property
def active_workers(self) -> int:
"""Number of worker tasks that are still alive."""
return sum(1 for w in self._workers if not w.done())
@property
def metrics(self) -> dict[str, int]:
"""Cumulative processing counters."""
m = self._metrics
return {
"total_processed": m.total_processed,
"total_successes": m.total_successes,
"total_failures": m.total_failures,
"total_timeouts": m.total_timeouts,
"pending": self.pending_count,
"active_workers": self.active_workers,
"chat_locks": len(self._chat_locks),
"sessions": len(self._sessions),
}
# ── ask_user helpers ──
async def _wait_for_ask_user_reply(
self,
session_key: str,
timeout: float,
) -> str | None:
"""Register a pending ask_user slot and wait for the user to reply.
Returns the raw reply text, or ``None`` on timeout.
"""
pending = _PendingAskUserReply(event=asyncio.Event())
self._pending_ask_user_replies[session_key] = pending
try:
await asyncio.wait_for(pending.event.wait(), timeout=timeout)
except TimeoutError:
pass
finally:
self._pending_ask_user_replies.pop(session_key, None)
return pending.reply
async def _resolve_ask_user(
self,
msg: InboundMessage,
event_data: dict,
session_key: str,
) -> dict:
"""Handle an ask_user interrupt: send questions to channel, collect answers.
Mirrors the logic of ``cli.channel.channel_ask_user_prompt`` but runs
fully async inside the consumer event loop.
Returns a dict suitable for ``Command(resume=...)``:
``{"answers": [...], "status": "answered"}`` or
``{"status": "cancelled"}``.
"""
questions = event_data.get("questions", [])
if not questions:
return {"answers": [], "status": "answered"}
total = len(questions)
answers: list[str] = []
for i, q in enumerate(questions):
q_text = q.get("question", "")
q_type = q.get("type", "text")
required = q.get("required", True)
# -- Format question header --
if total == 1:
header = "\u2753 Quick check-in from EvoScientist\n"
else:
header = f"\u2753 Question {i + 1}/{total}\n"
lines: list[str] = [header, f"{i + 1}. {q_text}"]
if not required:
lines[-1] += " (optional)"
if q_type == "multiple_choice":
choices = q.get("choices", [])
for j, choice in enumerate(choices):
label = choice.get("value", str(choice))
letter = chr(ord("A") + j)
lines.append(f" {letter}. {label}")
other_letter = chr(ord("A") + len(choices))
lines.append(f" {other_letter}. Other")
letters = "/".join(chr(ord("A") + k) for k in range(len(choices) + 1))
lines.append(f"\nReply with a letter ({letters}), or 'cancel'.")
else:
skip_hint = " Leave empty to skip." if not required else ""
lines.append(f"\nReply with your answer, or 'cancel'.{skip_hint}")
# -- Send question --
await self.bus.publish_outbound(
OutboundMessage(
channel=msg.channel,
chat_id=msg.chat_id,
content="\n".join(lines),
metadata=msg.metadata,
)
)
# -- Wait for user reply --
reply = await self._wait_for_ask_user_reply(
session_key,
_ASK_USER_TIMEOUT,
)
if not reply:
await self.bus.publish_outbound(
OutboundMessage(
channel=msg.channel,
chat_id=msg.chat_id,
content="\u23f0 Response timed out.",
metadata=msg.metadata,
)
)
return {"status": "cancelled"}
raw = reply.strip()
if raw.lower() == "cancel":
return {"status": "cancelled"}
# -- Parse answer --
if q_type == "multiple_choice":
choices = q.get("choices", [])
other_letter = chr(ord("A") + len(choices))
if len(raw) == 1 and raw.upper() == other_letter:
# "Other" selected — ask for free-form input
await self.bus.publish_outbound(
OutboundMessage(
channel=msg.channel,
chat_id=msg.chat_id,
content="Please type your answer:",
metadata=msg.metadata,
)
)
other_reply = await self._wait_for_ask_user_reply(
session_key,
_ASK_USER_TIMEOUT,
)
if not other_reply:
await self.bus.publish_outbound(
OutboundMessage(
channel=msg.channel,
chat_id=msg.chat_id,
content="\u23f0 Response timed out.",
metadata=msg.metadata,
)
)
return {"status": "cancelled"}
if other_reply.strip().lower() == "cancel":
return {"status": "cancelled"}
answers.append(other_reply.strip())
elif len(raw) == 1 and raw.upper().isalpha():
idx = ord(raw.upper()) - ord("A")
if 0 <= idx < len(choices):
answers.append(choices[idx].get("value", raw))
else:
answers.append(raw)
else:
answers.append(raw)
else:
answers.append(raw)
return {"answers": answers, "status": "answered"}
# ── internal ──
def _evict_chat_locks(self) -> None:
"""Remove chat locks that are not currently held."""
stale = [k for k, lock in self._chat_locks.items() if not lock.locked()]
for k in stale[: max(1, len(stale) // 2)]:
del self._chat_locks[k]
+174
View File
@@ -0,0 +1,174 @@
"""Shared debug logging helpers for channel integrations.
This module is intentionally channel-agnostic. Future per-channel PRs should
reuse these helpers instead of reintroducing ad-hoc logging formats or
standalone ``basicConfig`` calls.
"""
from __future__ import annotations
import logging
import os
from collections.abc import Mapping, Sequence
from typing import Any
_REDACTED = "***"
_SECRET_TOKENS = (
"token",
"secret",
"password",
"authorization",
"cookie",
"api_key",
"apikey",
"access_key",
"private_key",
"signature",
)
def _load_debug_trace_flag() -> bool:
"""Load the trace feature switch from config as a fallback to env vars."""
try:
from ..config.settings import load_config
return bool(getattr(load_config(), "channel_debug_tracing", False))
except Exception:
return False
def debug_trace_enabled(enabled: bool | None = None) -> bool:
"""Resolve the channel debug tracing switch.
Explicit ``enabled`` takes precedence; otherwise the helper falls back to
``EVOSCIENTIST_CHANNEL_DEBUG_TRACING``.
"""
if enabled is not None:
return bool(enabled)
raw = os.environ.get("EVOSCIENTIST_CHANNEL_DEBUG_TRACING", "")
if raw.strip():
return raw.strip().lower() in {"1", "true", "yes", "on"}
return _load_debug_trace_flag()
def _should_redact_key(key: str) -> bool:
lowered = key.lower()
return any(token in lowered for token in _SECRET_TOKENS)
def _stringify(value: Any) -> str:
if value is None:
return "-"
if isinstance(value, bool):
return "true" if value else "false"
if isinstance(value, bytes):
return f"<bytes:{len(value)}>"
if isinstance(value, str):
return value.replace("\n", "\\n")
if isinstance(value, Mapping):
return f"<map:{len(value)}>"
if isinstance(value, Sequence) and not isinstance(value, str | bytes | bytearray):
return f"<seq:{len(value)}>"
return str(value).replace("\n", "\\n")
def _format_fields(fields: Mapping[str, Any]) -> str:
parts: list[str] = []
for key, value in fields.items():
if value is None:
continue
safe_value = _REDACTED if _should_redact_key(key) else _stringify(value)
parts.append(f"{key}={safe_value}")
return " ".join(parts)
_warned_debug_level_mismatch = False
def _warn_debug_level_mismatch(logger: logging.Logger) -> None:
"""Emit a one-time warning when tracing is enabled but DEBUG logs are hidden."""
global _warned_debug_level_mismatch
if _warned_debug_level_mismatch:
return
_warned_debug_level_mismatch = True
logger.warning(
"channel debug tracing is enabled but logger level is above DEBUG; "
"set EVOSCIENTIST_LOG_LEVEL=DEBUG, configure log_level=debug, "
"or use 'serve --debug' to see trace events"
)
def emit_debug_event(
logger: logging.Logger,
event: str,
*,
channel: str,
enabled: bool,
**fields: Any,
) -> None:
"""Emit a structured channel debug event.
Example output:
``event=inbound_raw channel=telegram message_id=123 chat_id=-1001``
"""
if not enabled:
return
if not logger.isEnabledFor(logging.DEBUG):
_warn_debug_level_mismatch(logger)
return
base_fields = {"event": event, "channel": channel}
base_fields.update(fields)
logger.debug(_format_fields(base_fields))
def emit_debug_event_if(
logger: logging.Logger,
event: str,
enabled: bool,
**fields: Any,
) -> None:
"""Convenience wrapper for code that lacks a :class:`Channel` instance.
Unlike :func:`emit_debug_event`, the ``channel`` field is not required —
pass it via *fields* when available. This is intended for middleware
classes, managers, and standalone helpers.
"""
if not enabled:
return
if not logger.isEnabledFor(logging.DEBUG):
_warn_debug_level_mismatch(logger)
return
base_fields: dict[str, Any] = {"event": event}
base_fields.update(fields)
logger.debug(_format_fields(base_fields))
class TraceMixin:
"""Mixin providing unified structured trace helpers.
Classes using this mixin must set ``_debug_trace`` (bool) and
``_trace_logger`` attribute returning a :class:`logging.Logger`.
The trace name defaults to ``self.name`` if present, otherwise
``"unknown"``. Override ``_trace_name`` to customise.
"""
_debug_trace: bool
_trace_logger: logging.Logger
@property
def _trace_name(self) -> str:
return getattr(self, "name", "unknown")
def _trace_event(self, event: str, **fields: Any) -> None:
"""Emit a structured debug event when tracing is enabled."""
emit_debug_event(
self._trace_logger,
event,
channel=self._trace_name,
enabled=self._debug_trace,
**fields,
)
@@ -0,0 +1,31 @@
"""DingTalk (钉钉) channel for EvoScientist.
Uses Stream Mode (WebSocket) for receiving messages — no public IP needed.
Sends replies via HTTP API.
Usage in config:
channel_enabled = "dingtalk"
dingtalk_client_id = "your_app_key"
dingtalk_client_secret = "your_app_secret"
"""
from ..channel_manager import _parse_csv, register_channel
from .channel import DingTalkChannel, DingTalkConfig
__all__ = ["DingTalkChannel", "DingTalkConfig"]
def create_from_config(config) -> DingTalkChannel:
allowed = _parse_csv(config.dingtalk_allowed_senders)
proxy = config.dingtalk_proxy or None
return DingTalkChannel(
DingTalkConfig(
client_id=config.dingtalk_client_id,
client_secret=config.dingtalk_client_secret,
allowed_senders=allowed,
proxy=proxy,
)
)
register_channel("dingtalk", create_from_config)
+437
View File
@@ -0,0 +1,437 @@
"""DingTalk channel — refactored with WebSocketMixin + TokenMixin."""
import asyncio
import json
import logging
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import ClassVar
from urllib.parse import quote_plus
from ..base import Channel, ChannelError, RawIncoming
from ..capabilities import DINGTALK as DINGTALK_CAPS
from ..config import BaseChannelConfig
from ..mixins import TokenMixin, WebSocketMixin
logger = logging.getLogger(__name__)
GATEWAY_URL = "https://api.dingtalk.com/v1.0/gateway/connections/open"
TOKEN_URL = "https://api.dingtalk.com/v1.0/oauth2/accessToken"
SEND_URL = "https://api.dingtalk.com/v1.0/robot/oToMessages/batchSend"
MEDIA_SEND_URL = "https://api.dingtalk.com/v1.0/robot/oToMessages/batchSend"
MEDIA_UPLOAD_URL = "https://oapi.dingtalk.com/media/upload"
FILE_DOWNLOAD_URL = "https://api.dingtalk.com/v1.0/robot/messageFiles/download"
@dataclass
class DingTalkConfig(BaseChannelConfig):
client_id: str = ""
client_secret: str = ""
text_chunk_limit: int = 4096
class DingTalkChannel(Channel, WebSocketMixin, TokenMixin):
capabilities = DINGTALK_CAPS
name = "dingtalk"
_ready_attrs = ("_http_client", "_access_token")
_non_retryable_patterns = ("invalidauthentication", "forbidden", "40014")
_mention_pattern = r"@\S+\s*"
_mention_strip_count = 1
def __init__(self, config: DingTalkConfig):
super().__init__(config)
async def start(self) -> None:
import httpx
if not self.config.client_id or not self.config.client_secret:
raise ChannelError("DingTalk client_id and client_secret are required")
self._http_client = httpx.AsyncClient(timeout=15, proxy=self.config.proxy)
await self._refresh_token()
self._running = True
logger.info("DingTalk channel starting (Stream Mode)...")
self._ws_task = asyncio.create_task(self._ws_loop())
# ── TokenMixin ────────────────────────────────────────────────
async def _fetch_token(self) -> tuple[str, int]:
data = await self._api_post(
TOKEN_URL,
{
"appKey": self.config.client_id,
"appSecret": self.config.client_secret,
},
)
token = data.get("accessToken")
if not token:
raise ChannelError(f"DingTalk auth error: {data}")
return token, int(data.get("expireIn", 7200))
async def _api_post(self, url, body, headers=None):
resp = await self._http_client.post(url, json=body, headers=headers)
return resp.json()
async def _resolve_download_code(self, download_code: str) -> str | None:
"""Exchange a DingTalk downloadCode for a real download URL."""
try:
token = await self._ensure_token()
data = await self._api_post(
FILE_DOWNLOAD_URL,
{"downloadCode": download_code, "robotCode": self.config.client_id},
headers={"x-acs-dingtalk-access-token": token},
)
url = data.get("downloadUrl") or ""
if url:
return url
logger.warning(f"DingTalk downloadCode resolve failed: {data}")
except Exception as e:
logger.warning(f"DingTalk downloadCode resolve error: {e}")
return None
# ── WebSocketMixin ────────────────────────────────────────────
async def _get_ws_url(self) -> str:
resp = await self._http_client.post(
GATEWAY_URL,
json={
"clientId": self.config.client_id,
"clientSecret": self.config.client_secret,
"subscriptions": [
{"type": "CALLBACK", "topic": "/v1.0/im/bot/messages/get"}
],
"ua": "dingtalk-sdk-python/v0.24.3-union",
},
)
data = resp.json()
endpoint, ticket = data.get("endpoint"), data.get("ticket")
if not endpoint or not ticket:
raise ChannelError(f"DingTalk gateway failed: {data}")
return f"{endpoint}?ticket={quote_plus(ticket)}"
async def _on_ws_message(self, data) -> None:
if not isinstance(data, dict):
return
headers = data.get("headers", {})
msg_id = headers.get("messageId", "")
# System ping
if data.get("type") == "SYSTEM" and headers.get("topic") == "ping":
await self._ws_send_json(
{
"code": 200,
"headers": headers,
"message": "OK",
"data": data.get("data", ""),
}
)
return
# ACK
await self._ws_send_json(
{
"code": 200,
"headers": {"contentType": "application/json", "messageId": msg_id},
"message": "OK",
"data": "{}",
}
)
if data.get("type") != "CALLBACK":
return
payload = data.get("data", "{}")
payload = json.loads(payload) if isinstance(payload, str) else payload
text_obj = payload.get("text", {})
content = (
text_obj.get("content", "") if isinstance(text_obj, dict) else str(text_obj)
).strip()
if not content:
raw_content = payload.get("content", "")
content = raw_content.strip() if isinstance(raw_content, str) else ""
# Download attachments if present
annotations: list[str] = []
media_paths: list[str] = []
# DingTalk file/image messages may put download info in
# payload["content"] (as a dict) instead of in a dedicated
# "fileContent"/"imageContent" key.
raw_content_obj = payload.get("content")
if isinstance(raw_content_obj, dict) and raw_content_obj not in [
payload.get(k)
for k in ("imageContent", "fileContent", "videoContent", "audioContent")
]:
msg_type = payload.get("msgtype") or payload.get("msgType") or ""
media_label = msg_type or "file"
file_size = (
raw_content_obj.get("fileSize")
or raw_content_obj.get("downloadSize")
or 0
)
file_name = (
raw_content_obj.get("fileName")
or raw_content_obj.get("name")
or f"dingtalk_{msg_type}"
)
download_code = raw_content_obj.get("downloadCode") or ""
download_url = raw_content_obj.get("downloadUrl") or ""
# downloadCode is NOT a URL — resolve it via DingTalk API first
if download_code and not download_code.startswith("http"):
resolved = await self._resolve_download_code(download_code)
if resolved:
download_url = resolved
elif download_code:
download_url = download_code
if download_url:
try:
dl_token = await self._ensure_token()
dl_headers = {"x-acs-dingtalk-access-token": dl_token}
except Exception:
dl_headers = None
local, ann = await self._download_attachment(
download_url,
f"dingtalk_{file_name}",
headers=dl_headers,
file_size=int(file_size) if file_size else None,
)
if local:
media_paths.append(local)
if ann:
ann = ann.replace("[attachment:", f"[{media_label}:")
annotations.append(ann)
elif file_name:
annotations.append(f"[{media_label}: {file_name}]")
for att_key in ("imageContent", "fileContent", "videoContent", "audioContent"):
att = payload.get(att_key)
if att and isinstance(att, dict):
file_size = att.get("fileSize") or att.get("downloadSize") or 0
file_name = att.get("fileName", att_key)
download_code = att.get("downloadCode") or ""
download_url = att.get("downloadUrl") or ""
# Resolve downloadCode via API if it's not a URL
if download_code and not download_code.startswith("http"):
resolved = await self._resolve_download_code(download_code)
if resolved:
download_url = resolved
elif download_code:
download_url = download_code
# DingTalk audioContent is voice messages
media_label = "voice" if att_key == "audioContent" else att_key
if download_url and (
self.config.include_attachments
if hasattr(self.config, "include_attachments")
else True
):
# DingTalk download URLs require access token
try:
dl_token = await self._ensure_token()
dl_headers = {"x-acs-dingtalk-access-token": dl_token}
except Exception:
dl_headers = None
local, ann = await self._download_attachment(
download_url,
f"dingtalk_{file_name}",
headers=dl_headers,
file_size=int(file_size) if file_size else None,
)
if local:
media_paths.append(local)
if ann:
ann = ann.replace("[attachment:", f"[{media_label}:")
annotations.append(ann)
elif file_size:
too_large = self._check_attachment_size(int(file_size), file_name)
if too_large:
annotations.append(too_large)
else:
annotations.append(f"[{media_label}: {file_name}]")
if not content and not media_paths and not annotations:
return
sender_id = payload.get("senderStaffId") or payload.get("senderId", "")
is_group = payload.get("conversationType") == "2"
# For send API (oToMessages/batchSend), userIds needs staffId, not conversationId
chat_id = sender_id
create_time = payload.get("createAt") or payload.get("createTime", "")
# Mention gating: DMs always pass; groups require @bot
was_mentioned = not is_group
if is_group:
# isInAtList is set by DingTalk when bot is @mentioned
if payload.get("isInAtList"):
was_mentioned = True
else:
# Fallback: check atUsers array
at_users = payload.get("atUsers") or []
for u in at_users:
if u.get("dingtalkId") == self.config.client_id:
was_mentioned = True
break
try:
ts = (
datetime.fromtimestamp(int(create_time) / 1000)
if create_time
else datetime.now()
)
except (ValueError, TypeError, OSError):
ts = datetime.now()
await self._enqueue_raw(
RawIncoming(
sender_id=sender_id,
chat_id=chat_id,
text=content,
timestamp=ts,
message_id=msg_id,
is_group=is_group,
was_mentioned=was_mentioned,
media_files=media_paths,
content_annotations=annotations,
metadata={
"chat_id": chat_id,
"sender_nick": payload.get("senderNick", ""),
"backend": "dingtalk",
},
)
)
# _send_typing_action: inherited no-op (DingTalk has no typing API)
# _format_chunk: inherited from base (UnifiedFormatter)
# ── Send ──────────────────────────────────────────────────────
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
token = await self._ensure_token()
data = await self._api_post(
SEND_URL,
{
"robotCode": self.config.client_id,
"userIds": [chat_id],
"msgKey": "sampleMarkdown",
"msgParam": json.dumps({"text": raw_text, "title": "EvoScientist"}),
},
headers={"x-acs-dingtalk-access-token": token},
)
return data
# ── Media send ────────────────────────────────────────────────
_IMAGE_EXTS: ClassVar[set[str]] = {".jpg", ".jpeg", ".png", ".gif", ".bmp", ".webp"}
async def _send_media_impl(
self,
recipient: str,
file_path: str,
caption: str = "",
metadata: dict | None = None,
) -> bool:
"""Send a media file through DingTalk.
For images: uploads via /media/upload to get media_id, then sends
as sampleImageMsg. Non-image files are sent as markdown links
(DingTalk robot API does not support arbitrary file uploads).
"""
token = await self._ensure_token()
chat_id = self._resolve_media_chat_id(recipient, metadata)
headers = {"x-acs-dingtalk-access-token": token}
ext = Path(file_path).suffix.lower()
if ext in self._IMAGE_EXTS:
# Try uploading image to get media_id for native image message
media_id = await self._upload_dingtalk_media(token, file_path, "image")
if media_id:
await self._api_post(
MEDIA_SEND_URL,
{
"robotCode": self.config.client_id,
"userIds": [chat_id],
"msgKey": "sampleImageMsg",
"msgParam": json.dumps({"photoURL": media_id}),
},
headers=headers,
)
else:
# Fallback to markdown with file path
await self._api_post(
MEDIA_SEND_URL,
{
"robotCode": self.config.client_id,
"userIds": [chat_id],
"msgKey": "sampleMarkdown",
"msgParam": json.dumps(
{
"text": f"![image]({file_path})"
+ (f"\n{caption}" if caption else ""),
"title": caption or "Image",
}
),
},
headers=headers,
)
else:
# Non-image: send as markdown with filename
name = Path(file_path).name
text = f"[文件] {name}" + (f"\n{caption}" if caption else "")
await self._api_post(
MEDIA_SEND_URL,
{
"robotCode": self.config.client_id,
"userIds": [chat_id],
"msgKey": "sampleMarkdown",
"msgParam": json.dumps({"text": text, "title": name}),
},
headers=headers,
)
if caption and ext in self._IMAGE_EXTS:
# Send caption separately for image messages
await self._api_post(
MEDIA_SEND_URL,
{
"robotCode": self.config.client_id,
"userIds": [chat_id],
"msgKey": "sampleMarkdown",
"msgParam": json.dumps({"text": caption, "title": "Caption"}),
},
headers=headers,
)
return True
async def _upload_dingtalk_media(
self,
token: str,
file_path: str,
media_type: str = "image",
) -> str | None:
"""Upload a file to DingTalk media API and return the media_id."""
try:
url = f"{MEDIA_UPLOAD_URL}?access_token={token}&type={media_type}"
with open(file_path, "rb") as f:
resp = await self._http_client.post(
url,
files={"media": (Path(file_path).name, f)},
)
data = resp.json()
return data.get("media_id")
except Exception as e:
logger.warning(f"DingTalk media upload failed: {e}")
return None
async def _cleanup(self) -> None:
if hasattr(self, "_ws_task") and self._ws_task:
self._ws_task.cancel()
try:
await self._ws_task
except (asyncio.CancelledError, Exception):
pass
self._ws_task = None
await self._stop_ws()
if hasattr(self, "_http_client") and self._http_client:
await self._http_client.aclose()
self._http_client = None
self._access_token = None
logger.info("DingTalk channel stopped")
+33
View File
@@ -0,0 +1,33 @@
"""DingTalk credential validation."""
import logging
logger = logging.getLogger(__name__)
async def validate_dingtalk(
client_id: str,
client_secret: str,
proxy: str | None = None,
) -> tuple[bool, str]:
"""Validate DingTalk credentials by fetching an access token."""
if not client_id or not client_secret:
return False, "client_id and client_secret are required"
try:
import httpx
except ImportError:
return False, "httpx not installed"
url = "https://api.dingtalk.com/v1.0/oauth2/accessToken"
body = {"appKey": client_id, "appSecret": client_secret}
try:
async with httpx.AsyncClient(proxy=proxy) as client:
resp = await client.post(url, json=body, timeout=10)
data = resp.json()
if data.get("accessToken"):
return True, "DingTalk credentials valid"
return False, f"Error: {data.get('message', data)}"
except Exception as e:
return False, f"Error: {e}"
+89
View File
@@ -0,0 +1,89 @@
"""DingTalk channel server.
Standalone script to run the DingTalk channel with CLI options.
Usage:
python -m EvoScientist.channels.dingtalk.serve --client-id ID --client-secret SECRET [OPTIONS]
Examples:
# Basic usage
python -m EvoScientist.channels.dingtalk.serve --client-id ID --client-secret SECRET
# With proxy and allowed senders
python -m EvoScientist.channels.dingtalk.serve --client-id ID --client-secret SECRET --proxy http://proxy:8080 --allow user123
# With agent and thinking
python -m EvoScientist.channels.dingtalk.serve --client-id ID --client-secret SECRET --agent --thinking
"""
import argparse
import logging
from ...logging_config import configure_logging_from_settings
from ..bus import MessageBus
from ..standalone import run_standalone
from .channel import DingTalkChannel, DingTalkConfig
logger = logging.getLogger(__name__)
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="DingTalk channel server",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--client-id",
required=True,
help="DingTalk app client ID",
)
parser.add_argument(
"--client-secret",
required=True,
help="DingTalk app client secret",
)
parser.add_argument(
"--allow",
action="append",
dest="allowed_senders",
help="Allowed sender (DingTalk user ID). Can be used multiple times.",
)
parser.add_argument(
"--proxy",
help="HTTP proxy URL",
)
parser.add_argument(
"--agent",
action="store_true",
help="Use EvoScientist agent as handler (default: echo)",
)
parser.add_argument(
"--thinking",
action="store_true",
help="Send thinking content as intermediate messages (requires --agent)",
)
return parser.parse_args()
def main():
"""Entry point."""
configure_logging_from_settings(default_level=logging.INFO)
args = parse_args()
config = DingTalkConfig(
client_id=args.client_id,
client_secret=args.client_secret,
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
proxy=args.proxy,
)
send_thinking = args.thinking and args.agent
bus = MessageBus()
channel = DingTalkChannel(config)
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
if __name__ == "__main__":
main()
+21
View File
@@ -0,0 +1,21 @@
from ..channel_manager import _parse_csv, register_channel
from .channel import DiscordChannel, DiscordConfig
__all__ = ["DiscordChannel", "DiscordConfig"]
def create_from_config(config) -> DiscordChannel:
allowed = _parse_csv(config.discord_allowed_senders)
channels = _parse_csv(config.discord_allowed_channels)
proxy = config.discord_proxy or None
return DiscordChannel(
DiscordConfig(
bot_token=config.discord_bot_token,
allowed_senders=allowed,
allowed_channels=channels,
proxy=proxy,
)
)
register_channel("discord", create_from_config)
+274
View File
@@ -0,0 +1,274 @@
"""Discord channel implementation using discord.py."""
import asyncio
import logging
import os
from dataclasses import dataclass
from datetime import datetime
from ..base import Channel, ChannelError, RawIncoming
from ..capabilities import DISCORD as DISCORD_CAPS
from ..config import BaseChannelConfig
logger = logging.getLogger(__name__)
@dataclass
class DiscordConfig(BaseChannelConfig):
bot_token: str = ""
text_chunk_limit: int = 2000
class DiscordChannel(Channel):
"""Discord channel using discord.py."""
name = "discord"
capabilities = DISCORD_CAPS
_typing_interval: float = 8.0
_ready_attrs = ("_client",)
_mention_pattern = r"<@!?{bot_id}>\s*"
def __init__(self, config: DiscordConfig):
super().__init__(config)
self._client = None
self._ready = asyncio.Event()
# Cache message objects for ACK reactions
self._message_cache: dict[str, object] = {}
self._MESSAGE_CACHE_MAX = 200
self._background_tasks: set[asyncio.Task] = set()
async def start(self) -> None:
try:
import discord
except ImportError:
raise ChannelError(
"discord.py not installed. "
"Install with: pip install evoscientist[discord]"
) from None
if not self.config.bot_token:
raise ChannelError("Discord bot token is required")
proxy = (
self.config.proxy
or os.environ.get("https_proxy")
or os.environ.get("HTTPS_PROXY")
or os.environ.get("http_proxy")
or os.environ.get("HTTP_PROXY")
or None
)
logger.info(
"Discord connect: token=%s...%s proxy=%s",
self.config.bot_token[:8],
self.config.bot_token[-4:],
proxy or "(none)",
)
intents = discord.Intents.default()
intents.message_content = True
client_kwargs = {"intents": intents}
if proxy:
client_kwargs["proxy"] = proxy
self._client = discord.Client(**client_kwargs)
self._start_task_error: BaseException | None = None
@self._client.event
async def on_ready():
logger.info(f"Discord bot ready: {self._client.user}")
self._ready.set()
@self._client.event
async def on_message(message):
await self._on_message(message)
async def _guarded_start():
try:
logger.info("Discord gateway: starting client.start()...")
await self._client.start(self.config.bot_token)
except Exception as exc:
logger.error("Discord gateway error: %s: %s", type(exc).__name__, exc)
self._start_task_error = exc
self._ready.set() # unblock the waiter so it doesn't hang
logger.info("Discord connect: launching gateway task")
_task = asyncio.create_task(_guarded_start())
self._background_tasks.add(_task)
_task.add_done_callback(self._background_tasks.discard)
try:
await asyncio.wait_for(self._ready.wait(), timeout=60)
except TimeoutError:
raise ChannelError(
"Discord bot failed to connect within 60s. "
"Check network/proxy connectivity to gateway.discord.gg"
) from None
if self._start_task_error:
raise ChannelError(
f"Discord bot failed to connect: {self._start_task_error}"
)
self._running = True
logger.info("Discord channel started")
async def _cleanup(self) -> None:
if self._client:
await self._client.close()
logger.info("Discord channel stopped")
# ── Typing indicator ────────────────────────────────────────────
async def _send_typing_action(self, chat_id: str) -> None:
if not self._client:
return
ch = self._client.get_channel(int(chat_id))
if ch:
await ch.trigger_typing()
# ── ACK Reactions ───────────────────────────────────────────────
async def _send_ack_reaction(
self, chat_id: str, message_id: str, emoji: str = "👀"
) -> None:
msg = self._message_cache.get(message_id)
if msg:
try:
await msg.add_reaction(emoji)
except Exception as e:
logger.debug(f"Discord ACK reaction failed: {e}")
async def _remove_ack_reaction(
self, chat_id: str, message_id: str, emoji: str = "👀"
) -> None:
msg = self._message_cache.get(message_id)
if msg and self._client and self._client.user:
try:
await msg.remove_reaction(emoji, self._client.user)
except Exception as e:
logger.debug(f"Discord remove ACK reaction failed: {e}")
def _cache_message(self, message) -> None:
"""Cache a discord message object for later reaction use."""
mid = str(message.id)
self._message_cache[mid] = message
# Evict oldest entries if cache is too large
if len(self._message_cache) > self._MESSAGE_CACHE_MAX:
oldest = list(self._message_cache.keys())[: self._MESSAGE_CACHE_MAX // 2]
for k in oldest:
self._message_cache.pop(k, None)
# ── Send ────────────────────────────────────────────────────────
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
import discord
thread_id = (metadata or {}).get("thread_id", "")
target_id = int(thread_id) if thread_id else int(chat_id)
ch = self._client.get_channel(target_id)
if not ch:
raise RuntimeError(f"Discord channel {target_id} not found")
ref = None
if reply_to:
try:
ref = discord.MessageReference(
message_id=int(reply_to),
channel_id=target_id,
)
except (ValueError, TypeError):
pass
async def _send(text):
await ch.send(text, reference=ref)
await self._send_with_format_fallback(_send, formatted_text, raw_text)
async def _send_media_impl(
self,
recipient: str,
file_path: str,
caption: str = "",
metadata: dict | None = None,
) -> bool:
import discord
channel_id = self._resolve_media_chat_id(recipient, metadata)
ch = self._client.get_channel(int(channel_id))
if not ch:
logger.error(f"Discord channel {channel_id} not found")
return False
file = discord.File(file_path)
await ch.send(content=caption or None, file=file)
return True
def _get_bot_identifier(self) -> str | None:
if self._client and self._client.user:
return str(self._client.user.id)
return None
# ── Inbound ─────────────────────────────────────────────────────
async def _on_message(self, message) -> None:
import discord
if message.author == self._client.user:
return
# Cache for ACK reactions
self._cache_message(message)
user_id = str(message.author.id)
channel_id = str(message.channel.id)
is_dm = isinstance(message.channel, discord.DMChannel)
was_mentioned = is_dm or (self._client.user in message.mentions)
text = message.content or ""
annotations: list[str] = []
media_paths: list[str] = []
if self.config.include_attachments and message.attachments:
for attachment in message.attachments:
too_large = self._check_attachment_size(
attachment.size or 0,
attachment.filename,
)
if too_large:
annotations.append(too_large)
continue
try:
safe_name = attachment.filename.replace("/", "_")
file_path = self._media_path(f"{attachment.id}_{safe_name}")
await attachment.save(file_path)
media_paths.append(str(file_path))
annotations.append(f"[attachment: {file_path}]")
except Exception as e:
logger.warning(f"Failed to download Discord attachment: {e}")
annotations.append(
f"[attachment: {attachment.filename} - download failed]"
)
# Detect thread context
thread_id = ""
parent_channel_id = channel_id
if hasattr(message.channel, "parent") and message.channel.parent:
# Message is inside a Thread — store thread info
thread_id = channel_id # the thread IS the channel
parent_channel_id = str(message.channel.parent.id)
await self._enqueue_raw(
RawIncoming(
sender_id=user_id,
chat_id=parent_channel_id,
text=text,
media_files=media_paths,
content_annotations=annotations,
timestamp=message.created_at or datetime.now(),
message_id=str(message.id),
metadata={"chat_id": parent_channel_id, "thread_id": thread_id},
is_group=not is_dm,
was_mentioned=was_mentioned,
)
)
+35
View File
@@ -0,0 +1,35 @@
"""Discord bot token validation."""
import logging
logger = logging.getLogger(__name__)
async def validate_discord_token(
token: str, proxy: str | None = None
) -> tuple[bool, str]:
"""Validate a Discord bot token via the REST API.
Returns:
Tuple of (is_valid, message).
"""
if not token:
return False, "No token provided"
try:
import httpx
except ImportError:
return False, "httpx not installed"
url = "https://discord.com/api/v10/users/@me"
headers = {"Authorization": f"Bot {token}"}
try:
async with httpx.AsyncClient(proxy=proxy) as client:
resp = await client.get(url, headers=headers, timeout=10)
if resp.status_code == 200:
data = resp.json()
username = data.get("username", "unknown")
return True, f"Bot: {username}"
return False, "Invalid token"
except Exception as e:
return False, f"Error: {e}"
+90
View File
@@ -0,0 +1,90 @@
"""Discord channel server.
Standalone script to run the Discord channel with CLI options.
Usage:
python -m EvoScientist.channels.discord.serve --bot-token TOKEN [OPTIONS]
Examples:
# Allow all senders (default)
python -m EvoScientist.channels.discord.serve --bot-token TOKEN
# Only allow specific senders and channels
python -m EvoScientist.channels.discord.serve --bot-token TOKEN --allow 123 --allow-channel 456
# With proxy, agent and thinking
python -m EvoScientist.channels.discord.serve --bot-token TOKEN --proxy http://proxy:8080 --agent --thinking
"""
import argparse
import logging
from ...logging_config import configure_logging_from_settings
from ..bus import MessageBus
from ..standalone import run_standalone
from .channel import DiscordChannel, DiscordConfig
logger = logging.getLogger(__name__)
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="Discord channel server",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--bot-token",
required=True,
help="Discord bot token",
)
parser.add_argument(
"--allow",
action="append",
dest="allowed_senders",
help="Allowed sender (Discord user ID). Can be used multiple times.",
)
parser.add_argument(
"--allow-channel",
action="append",
dest="allowed_channels",
help="Allowed channel ID. Can be used multiple times.",
)
parser.add_argument(
"--proxy",
help="HTTP proxy URL for Discord API requests",
)
parser.add_argument(
"--agent",
action="store_true",
help="Use EvoScientist agent as handler (default: echo)",
)
parser.add_argument(
"--thinking",
action="store_true",
help="Send thinking content as intermediate messages (requires --agent)",
)
return parser.parse_args()
def main():
"""Entry point."""
configure_logging_from_settings(default_level=logging.INFO)
args = parse_args()
config = DiscordConfig(
bot_token=args.bot_token,
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
allowed_channels=set(args.allowed_channels) if args.allowed_channels else None,
proxy=args.proxy,
)
send_thinking = args.thinking and args.agent
bus = MessageBus()
channel = DiscordChannel(config)
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
if __name__ == "__main__":
main()
+43
View File
@@ -0,0 +1,43 @@
"""Email channel for EvoScientist.
Uses IMAP polling for inbound + SMTP for outbound. Pure Python, no extra deps.
Usage in config:
channel_enabled = "email"
email_imap_host = "imap.gmail.com"
email_smtp_host = "smtp.gmail.com"
...
"""
from ..channel_manager import _parse_csv, register_channel
from .channel import EmailChannel, EmailConfig
__all__ = ["EmailChannel", "EmailConfig"]
def create_from_config(config) -> EmailChannel:
allowed = _parse_csv(config.email_allowed_senders)
return EmailChannel(
EmailConfig(
imap_host=config.email_imap_host,
imap_port=config.email_imap_port,
imap_username=config.email_imap_username,
imap_password=config.email_imap_password,
imap_mailbox=config.email_imap_mailbox,
imap_use_ssl=config.email_imap_use_ssl,
smtp_host=config.email_smtp_host,
smtp_port=config.email_smtp_port,
smtp_username=config.email_smtp_username,
smtp_password=config.email_smtp_password,
smtp_starttls=config.email_smtp_use_tls,
from_address=config.email_from_address,
poll_interval=config.email_poll_interval,
mark_seen=config.email_mark_seen,
max_body_chars=config.email_max_body_chars,
subject_prefix=config.email_subject_prefix,
allowed_senders=allowed,
)
)
register_channel("email", create_from_config)
+454
View File
@@ -0,0 +1,454 @@
"""Email channel implementation using IMAP + SMTP."""
import asyncio
import contextlib
import email as email_lib
import email.utils
import html
import imaplib
import logging
import re
import smtplib
import ssl
from dataclasses import dataclass
from datetime import datetime
from email import encoders
from email.header import decode_header, make_header
from email.message import EmailMessage
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.utils import parseaddr
from pathlib import Path
from ..base import Channel, ChannelError, RawIncoming
from ..capabilities import EMAIL as EMAIL_CAPS
from ..config import BaseChannelConfig
from ..mixins import PollingMixin
logger = logging.getLogger(__name__)
def _decode_hdr(raw: str) -> str:
try:
return str(make_header(decode_header(raw))) if raw else ""
except Exception:
return raw or ""
def _strip_html(text: str) -> str:
text = re.sub(r"<br\s*/?>", "\n", text, flags=re.IGNORECASE)
text = re.sub(r"<p[^>]*>", "\n", text, flags=re.IGNORECASE)
text = re.sub(r"</p>", "\n", text, flags=re.IGNORECASE)
text = re.sub(r"<[^>]+>", "", text)
return html.unescape(text).strip()
@dataclass
class EmailConfig(BaseChannelConfig):
imap_host: str = ""
imap_port: int = 993
imap_username: str = ""
imap_password: str = ""
imap_mailbox: str = "INBOX"
imap_use_ssl: bool = True
smtp_host: str = ""
smtp_port: int = 587
smtp_username: str = ""
smtp_password: str = ""
smtp_starttls: bool = (
True # True=STARTTLS (port 587), False=implicit SSL (port 465)
)
from_address: str = ""
poll_interval: int = 30
mark_seen: bool = True
max_body_chars: int = 12000
subject_prefix: str = "Re: "
text_chunk_limit: int = 4096
class EmailChannel(Channel, PollingMixin):
"""Email channel using IMAP polling + SMTP."""
name = "email"
capabilities = EMAIL_CAPS
_non_retryable_patterns = ("auth", "login", "credential")
def __init__(self, config: EmailConfig):
super().__init__(config)
self._imap: imaplib.IMAP4_SSL | imaplib.IMAP4 | None = None
async def start(self) -> None:
cfg = self.config
if not cfg.imap_host or not cfg.imap_username:
raise ChannelError("Email imap_host and imap_username are required")
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, self._connect_imap)
self._running = True
logger.info(
f"Email channel started (IMAP: {cfg.imap_host}, poll {cfg.poll_interval}s)"
)
await self._start_polling()
async def _cleanup(self) -> None:
await self._stop_polling()
if self._imap:
try:
self._imap.close()
self._imap.logout()
except Exception:
pass
self._imap = None
logger.info("Email channel stopped")
def _connect_imap(self) -> None:
cfg = self.config
try:
if cfg.imap_use_ssl:
self._imap = imaplib.IMAP4_SSL(
cfg.imap_host,
cfg.imap_port,
ssl_context=ssl.create_default_context(),
)
else:
self._imap = imaplib.IMAP4(cfg.imap_host, cfg.imap_port)
self._imap.login(cfg.imap_username, cfg.imap_password)
self._imap.select(cfg.imap_mailbox)
except Exception as e:
raise ChannelError(f"IMAP failed: {e}") from e
def _reconnect_imap(self) -> None:
try:
if self._imap:
self._imap.noop()
return
except Exception:
pass
self._connect_imap()
async def _poll_once(self) -> None:
loop = asyncio.get_running_loop()
messages = await loop.run_in_executor(None, self._fetch_unseen)
for m in messages:
await self._process_email(m)
def _fetch_unseen(self) -> list[dict]:
self._reconnect_imap()
results = []
try:
st, data = self._imap.search(None, "UNSEEN")
if st != "OK":
return []
for mid in data[0].split()[-20:]:
st, msg_data = self._imap.fetch(mid, "(RFC822)")
if st != "OK":
continue
msg = email_lib.message_from_bytes(msg_data[0][1])
from_name, from_addr = parseaddr(msg.get("From", ""))
body = self._extract_body(msg)
if len(body) > self.config.max_body_chars:
body = body[: self.config.max_body_chars] + "\n[...truncated]"
# Extract attachments and inline images
attachments = []
if msg.is_multipart():
for part in msg.walk():
content_disp = part.get("Content-Disposition") or ""
content_type = part.get_content_type() or ""
is_attachment = "attachment" in content_disp.lower()
is_inline_image = (
"inline" in content_disp.lower()
and content_type.startswith("image/")
)
# Also detect non-text parts with a filename but no
# Content-Disposition header (common for PDFs, docs,
# etc. sent by some email clients).
is_named_file = (
not is_attachment
and not is_inline_image
and part.get_filename()
and not content_type.startswith("multipart/")
and not content_type.startswith("text/")
)
if is_attachment or is_inline_image or is_named_file:
filename = part.get_filename() or "attachment"
filename = _decode_hdr(filename)
payload_data = part.get_payload(decode=True)
if payload_data:
from ..base import MAX_ATTACHMENT_BYTES, MEDIA_DIR
if len(payload_data) > MAX_ATTACHMENT_BYTES:
attachments.append(
{
"annotation": f"[attachment: {filename} - too large ({len(payload_data)} bytes)]"
}
)
else:
MEDIA_DIR.mkdir(parents=True, exist_ok=True)
local_path = (
MEDIA_DIR / f"email_{mid.decode()}_{filename}"
)
local_path.write_bytes(payload_data)
label = (
"inline-image"
if is_inline_image
else "attachment"
)
attachments.append(
{
"path": str(local_path),
"annotation": f"[{label}: {local_path}]",
}
)
if self.config.mark_seen:
self._imap.store(mid, "+FLAGS", "\\Seen")
results.append(
{
"from_addr": from_addr,
"from_name": _decode_hdr(from_name),
"subject": _decode_hdr(msg.get("Subject", "")),
"body": body,
"message_id": msg.get("Message-ID", ""),
"date": msg.get("Date", ""),
"references": msg.get("References", ""),
"attachments": attachments,
}
)
except Exception as e:
logger.error(f"IMAP fetch: {e}")
return results
def _extract_body(self, msg) -> str:
if msg.is_multipart():
for part in msg.walk():
ct = part.get_content_type()
if ct == "text/plain":
return self._decode_payload(part)
for part in msg.walk():
if part.get_content_type() == "text/html":
return _strip_html(self._decode_payload(part))
return "[no text content]"
text = self._decode_payload(msg)
return _strip_html(text) if msg.get_content_type() == "text/html" else text
@staticmethod
def _decode_payload(part) -> str:
payload = part.get_payload(decode=True)
if not payload:
return ""
charset = part.get_content_charset() or "utf-8"
return payload.decode(charset, errors="replace")
async def _process_email(self, m: dict) -> None:
subject = m["subject"]
text = f"[邮件] 主题: {subject}\n\n{m['body']}" if subject else m["body"]
try:
ts = email_lib.utils.parsedate_to_datetime(m["date"])
except Exception:
ts = datetime.now()
# Process attachments
media_paths: list[str] = []
annotations: list[str] = []
for att in m.get("attachments", []):
if att.get("path"):
media_paths.append(att["path"])
if att.get("annotation"):
annotations.append(att["annotation"])
await self._enqueue_raw(
RawIncoming(
sender_id=m["from_addr"],
chat_id=m["from_addr"],
text=text,
timestamp=ts,
message_id=m["message_id"],
media_files=media_paths,
content_annotations=annotations,
metadata={
"chat_id": m["from_addr"],
"subject": subject,
"original_message_id": m["message_id"],
"references": m["references"],
"backend": "email",
},
)
)
# ── Send ──────────────────────────────────────────────────────
def _is_ready(self) -> bool:
return bool(self.config.smtp_host)
@contextlib.contextmanager
def _smtp_connect(self):
"""Open an SMTP connection as a context manager.
Ensures the connection is closed even if login or send raises.
Uses STARTTLS (port 587) when smtp_starttls=True, otherwise
implicit SSL (port 465).
"""
cfg = self.config
srv = None
try:
if cfg.smtp_starttls:
srv = smtplib.SMTP(cfg.smtp_host, cfg.smtp_port, timeout=30)
srv.starttls()
else:
srv = smtplib.SMTP_SSL(
cfg.smtp_host,
cfg.smtp_port,
context=ssl.create_default_context(),
timeout=30,
)
srv.login(cfg.smtp_username, cfg.smtp_password)
yield srv
finally:
if srv is not None:
try:
srv.quit()
except Exception:
try:
srv.close()
except Exception:
pass
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
loop = asyncio.get_running_loop()
try:
await loop.run_in_executor(
None,
self._smtp_send_html,
chat_id,
formatted_text,
raw_text,
metadata or {},
)
except Exception as e:
err_str = str(e).lower()
# Only fall back to plain text for format-related errors, not server rejections
if any(
code in err_str for code in ("550", "553", "554", "auth", "rejected")
):
raise
logger.warning(f"HTML email failed ({e}), falling back to plain text")
await loop.run_in_executor(
None,
self._smtp_send,
chat_id,
raw_text,
metadata or {},
)
def _smtp_send(self, to: str, content: str, meta: dict) -> None:
cfg = self.config
from_addr = cfg.from_address or cfg.smtp_username
logger.debug(f"SMTP plain send: from={from_addr} to={to}")
msg = EmailMessage()
orig_subj = meta.get("subject", "")
msg["Subject"] = (
f"{cfg.subject_prefix}{orig_subj}"
if orig_subj and not orig_subj.lower().startswith("re:")
else (orig_subj or "EvoScientist Reply")
)
msg["From"] = from_addr
msg["To"] = to
orig_id = meta.get("original_message_id", "")
if orig_id:
msg["In-Reply-To"] = orig_id
msg["References"] = f"{meta.get('references', '')} {orig_id}".strip()
msg.set_content(content)
try:
with self._smtp_connect() as srv:
srv.sendmail(from_addr, [to], msg.as_string())
except Exception as e:
logger.error(f"SMTP send failed: from={from_addr} to={to}")
raise RuntimeError("SMTP send failed") from e
def _smtp_send_html(
self, to: str, html_content: str, plain_content: str, meta: dict
) -> None:
"""Send an email with both HTML and plain-text parts."""
cfg = self.config
from_addr = cfg.from_address or cfg.smtp_username
logger.debug(f"SMTP HTML send: from={from_addr} to={to}")
msg = MIMEMultipart("alternative")
orig_subj = meta.get("subject", "")
msg["Subject"] = (
f"{cfg.subject_prefix}{orig_subj}"
if orig_subj and not orig_subj.lower().startswith("re:")
else (orig_subj or "EvoScientist Reply")
)
msg["From"] = from_addr
msg["To"] = to
orig_id = meta.get("original_message_id", "")
if orig_id:
msg["In-Reply-To"] = orig_id
msg["References"] = f"{meta.get('references', '')} {orig_id}".strip()
msg.attach(MIMEText(plain_content, "plain", "utf-8"))
msg.attach(MIMEText(html_content, "html", "utf-8"))
try:
with self._smtp_connect() as srv:
srv.sendmail(from_addr, [to], msg.as_string())
except Exception as e:
logger.error(f"SMTP HTML send failed: from={from_addr} to={to}")
raise RuntimeError("SMTP HTML send failed") from e
# ── Media send (email attachment) ─────────────────────────────
async def _send_media_impl(
self,
recipient: str,
file_path: str,
caption: str = "",
metadata: dict | None = None,
) -> bool:
"""Send a file as an email attachment via SMTP."""
loop = asyncio.get_running_loop()
await loop.run_in_executor(
None,
self._smtp_send_attachment,
recipient,
file_path,
caption,
metadata or {},
)
return True
def _smtp_send_attachment(
self, to: str, file_path: str, caption: str, meta: dict
) -> None:
"""Send an email with a file attachment."""
cfg = self.config
from_addr = cfg.from_address or cfg.smtp_username
logger.debug(f"SMTP attachment send: from={from_addr} to={to} file={file_path}")
msg = MIMEMultipart()
orig_subj = meta.get("subject", "")
msg["Subject"] = (
f"{cfg.subject_prefix}{orig_subj}"
if orig_subj and not orig_subj.lower().startswith("re:")
else (orig_subj or "EvoScientist Reply")
)
msg["From"] = from_addr
msg["To"] = to
orig_id = meta.get("original_message_id", "")
if orig_id:
msg["In-Reply-To"] = orig_id
msg["References"] = f"{meta.get('references', '')} {orig_id}".strip()
# Text body
if caption:
msg.attach(MIMEText(caption, "plain", "utf-8"))
# Attachment
path = Path(file_path)
part = MIMEBase("application", "octet-stream")
part.set_payload(path.read_bytes())
encoders.encode_base64(part)
part.add_header("Content-Disposition", f"attachment; filename={path.name}")
msg.attach(part)
try:
with self._smtp_connect() as srv:
srv.sendmail(from_addr, [to], msg.as_string())
except Exception as e:
logger.error(f"SMTP attachment send failed: from={from_addr} to={to}")
raise RuntimeError("SMTP attachment send failed") from e
+92
View File
@@ -0,0 +1,92 @@
"""Email credential validation."""
import imaplib
import logging
import smtplib
import ssl
logger = logging.getLogger(__name__)
async def validate_email_imap(
host: str,
port: int,
username: str,
password: str,
use_ssl: bool = True,
) -> tuple[bool, str]:
"""Validate IMAP credentials.
Returns:
Tuple of (is_valid, message).
"""
if not host or not username or not password:
return False, "host, username, and password are required"
import asyncio
loop = asyncio.get_event_loop()
def _check():
try:
if use_ssl:
ctx = ssl.create_default_context()
conn = imaplib.IMAP4_SSL(host, port, ssl_context=ctx)
else:
conn = imaplib.IMAP4(host, port)
conn.login(username, password)
conn.logout()
return True, "IMAP credentials valid"
except imaplib.IMAP4.error as e:
return False, f"IMAP auth failed: {e}"
except Exception as e:
return False, f"IMAP error: {e}"
return await loop.run_in_executor(None, _check)
async def validate_email_smtp(
host: str,
port: int,
username: str,
password: str,
use_tls: bool = True,
) -> tuple[bool, str]:
"""Validate SMTP credentials.
Returns:
Tuple of (is_valid, message).
"""
if not host or not username or not password:
return False, "host, username, and password are required"
import asyncio
loop = asyncio.get_event_loop()
def _check():
server = None
try:
if use_tls:
server = smtplib.SMTP(host, port, timeout=10)
server.starttls()
else:
ctx = ssl.create_default_context()
server = smtplib.SMTP_SSL(host, port, context=ctx, timeout=10)
server.login(username, password)
return True, "SMTP credentials valid"
except smtplib.SMTPAuthenticationError:
return False, "SMTP auth failed"
except Exception as e:
return False, f"SMTP error: {e}"
finally:
if server is not None:
try:
server.quit()
except Exception:
try:
server.close()
except Exception:
pass
return await loop.run_in_executor(None, _check)
+121
View File
@@ -0,0 +1,121 @@
"""Email channel server.
Standalone script to run the Email channel with CLI options.
Usage:
python -m EvoScientist.channels.email.serve --imap-host HOST --imap-username USER --imap-password PASS --smtp-host HOST --smtp-username USER --smtp-password PASS --from-address ADDR [OPTIONS]
Examples:
# Basic usage
python -m EvoScientist.channels.email.serve --imap-host imap.gmail.com --imap-username bot@gmail.com --imap-password PASS --smtp-host smtp.gmail.com --smtp-username bot@gmail.com --smtp-password PASS --from-address bot@gmail.com
# With allowed senders and custom poll interval
python -m EvoScientist.channels.email.serve --imap-host imap.gmail.com --imap-username bot@gmail.com --imap-password PASS --smtp-host smtp.gmail.com --smtp-username bot@gmail.com --smtp-password PASS --from-address bot@gmail.com --allow user@example.com --poll-interval 60
# With agent and thinking
python -m EvoScientist.channels.email.serve --imap-host imap.gmail.com --imap-username bot@gmail.com --imap-password PASS --smtp-host smtp.gmail.com --smtp-username bot@gmail.com --smtp-password PASS --from-address bot@gmail.com --agent --thinking
"""
import argparse
import logging
from ...logging_config import configure_logging_from_settings
from ..bus import MessageBus
from ..standalone import run_standalone
from .channel import EmailChannel, EmailConfig
logger = logging.getLogger(__name__)
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="Email channel server",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--imap-host",
required=True,
help="IMAP server hostname",
)
parser.add_argument(
"--imap-username",
required=True,
help="IMAP username",
)
parser.add_argument(
"--imap-password",
required=True,
help="IMAP password",
)
parser.add_argument(
"--smtp-host",
required=True,
help="SMTP server hostname",
)
parser.add_argument(
"--smtp-username",
required=True,
help="SMTP username",
)
parser.add_argument(
"--smtp-password",
required=True,
help="SMTP password",
)
parser.add_argument(
"--from-address",
required=True,
help="From email address for outgoing messages",
)
parser.add_argument(
"--allow",
action="append",
dest="allowed_senders",
help="Allowed sender (email address). Can be used multiple times.",
)
parser.add_argument(
"--poll-interval",
type=int,
default=30,
help="IMAP poll interval in seconds (default: 30)",
)
parser.add_argument(
"--agent",
action="store_true",
help="Use EvoScientist agent as handler (default: echo)",
)
parser.add_argument(
"--thinking",
action="store_true",
help="Send thinking content as intermediate messages (requires --agent)",
)
return parser.parse_args()
def main():
"""Entry point."""
configure_logging_from_settings(default_level=logging.INFO)
args = parse_args()
config = EmailConfig(
imap_host=args.imap_host,
imap_username=args.imap_username,
imap_password=args.imap_password,
smtp_host=args.smtp_host,
smtp_username=args.smtp_username,
smtp_password=args.smtp_password,
from_address=args.from_address,
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
poll_interval=args.poll_interval,
)
send_thinking = args.thinking and args.agent
bus = MessageBus()
channel = EmailChannel(config)
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
if __name__ == "__main__":
main()
+25
View File
@@ -0,0 +1,25 @@
from ..channel_manager import _parse_csv, register_channel
from .channel import FeishuChannel, FeishuConfig
__all__ = ["FeishuChannel", "FeishuConfig"]
def create_from_config(config) -> FeishuChannel:
allowed = _parse_csv(config.feishu_allowed_senders)
proxy = config.feishu_proxy or None
return FeishuChannel(
FeishuConfig(
app_id=config.feishu_app_id,
app_secret=config.feishu_app_secret,
verification_token=config.feishu_verification_token,
encrypt_key=config.feishu_encrypt_key,
webhook_port=config.feishu_webhook_port,
allowed_senders=allowed,
feishu_domain=config.feishu_domain,
proxy=proxy,
subscription_mode=getattr(config, "feishu_subscription_mode", "webhook"),
)
)
register_channel("feishu", create_from_config)
File diff suppressed because it is too large Load Diff
+39
View File
@@ -0,0 +1,39 @@
"""Feishu (飞书/Lark) app credential validation."""
import logging
logger = logging.getLogger(__name__)
async def validate_feishu_credentials(
app_id: str,
app_secret: str,
domain: str = "https://open.feishu.cn",
) -> tuple[bool, str]:
"""Validate Feishu app credentials by requesting a tenant_access_token.
Returns:
Tuple of (is_valid, message).
"""
if not app_id:
return False, "No app_id provided"
if not app_secret:
return False, "No app_secret provided"
try:
import httpx
except ImportError:
return False, "httpx not installed"
url = f"{domain}/open-apis/auth/v3/tenant_access_token/internal"
body = {"app_id": app_id, "app_secret": app_secret}
try:
async with httpx.AsyncClient() as client:
resp = await client.post(url, json=body, timeout=10)
data = resp.json()
if data.get("code") == 0:
return True, f"App: {app_id}"
msg = data.get("msg", "unknown error")
return False, f"Auth failed: {msg}"
except Exception as e:
return False, f"Error: {e}"
+117
View File
@@ -0,0 +1,117 @@
"""Feishu (飞书/Lark) channel server.
Standalone script to run the Feishu channel with CLI options.
Usage:
python -m EvoScientist.channels.feishu.serve --app-id ID --app-secret SECRET [OPTIONS]
Examples:
# Basic setup
python -m EvoScientist.channels.feishu.serve --app-id ID --app-secret SECRET
# With verification token and custom port
python -m EvoScientist.channels.feishu.serve --app-id ID --app-secret SECRET \\
--verification-token TOKEN --webhook-port 9000
# With agent and thinking
python -m EvoScientist.channels.feishu.serve --app-id ID --app-secret SECRET --agent --thinking
"""
import argparse
import logging
from ...logging_config import configure_logging_from_settings
from ..bus import MessageBus
from ..standalone import run_standalone
from .channel import FeishuChannel, FeishuConfig
logger = logging.getLogger(__name__)
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="Feishu (飞书/Lark) channel server",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--app-id",
required=True,
help="Feishu App ID",
)
parser.add_argument(
"--app-secret",
required=True,
help="Feishu App Secret",
)
parser.add_argument(
"--verification-token",
default="",
help="Feishu event verification token",
)
parser.add_argument(
"--encrypt-key",
default="",
help="Feishu event encrypt key",
)
parser.add_argument(
"--webhook-port",
type=int,
default=9000,
help="Port for webhook HTTP server (default: 9000)",
)
parser.add_argument(
"--domain",
default="https://open.feishu.cn",
help="Feishu API domain (use https://open.larksuite.com for Lark)",
)
parser.add_argument(
"--allow",
action="append",
dest="allowed_senders",
help="Allowed sender (Feishu open_id). Can be used multiple times.",
)
parser.add_argument(
"--mode",
choices=["webhook", "websocket"],
default="webhook",
help="Subscription mode: webhook (default) or websocket (long connection, no public IP needed)",
)
parser.add_argument(
"--agent",
action="store_true",
help="Use EvoScientist agent as handler (default: echo)",
)
parser.add_argument(
"--thinking",
action="store_true",
help="Send thinking content as intermediate messages (requires --agent)",
)
return parser.parse_args()
def main():
"""Entry point."""
configure_logging_from_settings(default_level=logging.INFO)
args = parse_args()
config = FeishuConfig(
app_id=args.app_id,
app_secret=args.app_secret,
verification_token=args.verification_token,
encrypt_key=args.encrypt_key,
webhook_port=args.webhook_port,
feishu_domain=args.domain,
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
subscription_mode=args.mode,
)
send_thinking = args.thinking and args.agent
bus = MessageBus()
channel = FeishuChannel(config)
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
if __name__ == "__main__":
main()
+295
View File
@@ -0,0 +1,295 @@
"""Unified formatting pipeline for all channels.
Internal representation is Markdown. This module converts Markdown to
each platform's native format: HTML, Slack mrkdwn, Discord Markdown,
or plain text.
Channels no longer need per-file format functions — they just declare
``capabilities.format_type`` and the base class auto-configures a
``UnifiedFormatter`` instance.
"""
from __future__ import annotations
import re
from collections.abc import Callable
from typing import ClassVar
# ═════════════════════════════════════════════════════════════════════
# Markdown conversion engine (formerly markdown_utils.py)
# ═════════════════════════════════════════════════════════════════════
_PLACEHOLDER_PREFIX = "\x00BLOCK"
_INLINE_PREFIX = "\x00INLINE"
# A formatting rule: (regex_pattern, replacement)
InlineRule = tuple[str, str]
def convert_markdown(
text: str,
*,
code_block_formatter: Callable[[str, str], str],
inline_code_formatter: Callable[[str], str],
inline_rules: list[InlineRule],
escape_fn: Callable[[str], str] | None = None,
) -> str:
"""Convert Markdown to a channel-specific format.
Parameters
----------
text:
Input Markdown text.
code_block_formatter:
``(language, code) -> str`` — format a fenced code block.
inline_code_formatter:
``(code) -> str`` — format an inline code span.
inline_rules:
List of ``(pattern, replacement)`` pairs applied in order to the
remaining text (after code extraction and optional escaping).
escape_fn:
Optional function applied to the non-code text *before* inline
rules. Useful for HTML-escaping (Telegram) or other channel-
specific character escaping.
Returns
-------
str
The converted text.
"""
# 1. Extract and protect fenced code blocks (```...```)
code_blocks: list[str] = []
def _save_code_block(m: re.Match) -> str:
lang = m.group(1) or ""
code = m.group(2)
formatted = code_block_formatter(lang, code)
idx = len(code_blocks)
code_blocks.append(formatted)
return f"{_PLACEHOLDER_PREFIX}{idx}\x00"
text = re.sub(r"```(\w*)\n?(.*?)```", _save_code_block, text, flags=re.DOTALL)
# 2. Extract and protect inline code (`...`)
inline_codes: list[str] = []
def _save_inline(m: re.Match) -> str:
code = m.group(1)
formatted = inline_code_formatter(code)
idx = len(inline_codes)
inline_codes.append(formatted)
return f"{_INLINE_PREFIX}{idx}\x00"
text = re.sub(r"`([^`]+)`", _save_inline, text)
# 3. Optional escaping of remaining text
if escape_fn is not None:
text = escape_fn(text)
# 4. Apply inline formatting rules
for pattern, replacement in inline_rules:
text = re.sub(pattern, replacement, text, flags=re.MULTILINE)
# 5. Restore code blocks and inline code
for idx, html in enumerate(code_blocks):
text = text.replace(f"{_PLACEHOLDER_PREFIX}{idx}\x00", html)
for idx, code in enumerate(inline_codes):
text = text.replace(f"{_INLINE_PREFIX}{idx}\x00", code)
return text
# ═════════════════════════════════════════════════════════════════════
# Shared helpers
# ═════════════════════════════════════════════════════════════════════
def _escape_html(text: str) -> str:
return text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
def _noop_escape(text: str) -> str:
return text
# ═════════════════════════════════════════════════════════════════════
# HTML profile (Telegram, Email, Teams)
# ═════════════════════════════════════════════════════════════════════
def _html_code_block(lang: str, code: str) -> str:
escaped = _escape_html(code)
if lang:
return f'<pre><code class="language-{lang}">{escaped}</code></pre>'
return f"<pre><code>{escaped}</code></pre>"
def _html_inline_code(code: str) -> str:
return f"<code>{_escape_html(code)}</code>"
_HTML_INLINE_RULES: list[InlineRule] = [
# Headings → bold
(r"^#{1,6}\s+(.+)$", r"<b>\1</b>"),
# Blockquote markers (already escaped to &gt;)
(r"^&gt;\s?", ""),
# Links [text](url) → <a>
(r"\[([^\]]+)\]\(([^)]+)\)", r'<a href="\2">\1</a>'),
# Bold **text** → <b>
(r"\*\*(.+?)\*\*", r"<b>\1</b>"),
# Italic _text_ → <i>
(r"(?<!\w)_([^_]+?)_(?!\w)", r"<i>\1</i>"),
# Strikethrough ~~text~~ → <s>
(r"~~(.+?)~~", r"<s>\1</s>"),
# List items
(r"^[\-\*]\s+", "• "),
]
# ═════════════════════════════════════════════════════════════════════
# Slack mrkdwn profile
# ═════════════════════════════════════════════════════════════════════
def _slack_code_block(lang: str, code: str) -> str:
return f"```\n{code}```"
def _slack_inline_code(code: str) -> str:
return f"`{code}`"
_SLACK_INLINE_RULES: list[InlineRule] = [
(r"^#{1,6}\s+(.+)$", r"*\1*"),
(r"\[([^\]]+)\]\(([^)]+)\)", r"<\2|\1>"),
(r"\*\*(.+?)\*\*", r"*\1*"),
(r"~~(.+?)~~", r"~\1~"),
(r"^[\-\*]\s+", "• "),
]
# ═════════════════════════════════════════════════════════════════════
# Discord profile (mostly passthrough, headings → bold)
# ═════════════════════════════════════════════════════════════════════
def _discord_code_block(lang: str, code: str) -> str:
return f"```{lang}\n{code}```"
def _discord_inline_code(code: str) -> str:
return f"`{code}`"
_DISCORD_INLINE_RULES: list[InlineRule] = [
(r"^#{1,6}\s+(.+)$", r"**\1**"),
]
# ═════════════════════════════════════════════════════════════════════
# Plain text profile (strip all formatting)
# ═════════════════════════════════════════════════════════════════════
def _plain_code_block(lang: str, code: str) -> str:
return code
def _plain_inline_code(code: str) -> str:
return code
_PLAIN_INLINE_RULES: list[InlineRule] = [
(r"^#{1,6}\s+", ""),
(r"\[([^\]]+)\]\(([^)]+)\)", r"\1 (\2)"),
(r"\*\*(.+?)\*\*", r"\1"),
(r"(?<!\w)_([^_]+?)_(?!\w)", r"\1"),
(r"~~(.+?)~~", r"\1"),
(r"^[\-\*]\s+", "• "),
]
# ═════════════════════════════════════════════════════════════════════
# Markdown passthrough profile (Feishu, DingTalk, WeCom)
# ═════════════════════════════════════════════════════════════════════
def _md_code_block(lang: str, code: str) -> str:
return f"```{lang}\n{code}```"
def _md_inline_code(code: str) -> str:
return f"`{code}`"
_MD_INLINE_RULES: list[InlineRule] = [] # passthrough — already Markdown
# ═════════════════════════════════════════════════════════════════════
# Unified Formatter
# ═════════════════════════════════════════════════════════════════════
class UnifiedFormatter:
"""Converts internal Markdown to a target platform format.
Instantiated once per channel based on its ``capabilities.format_type``.
"""
_PROFILES: ClassVar[dict[str, dict]] = {
"html": {
"code_block_formatter": _html_code_block,
"inline_code_formatter": _html_inline_code,
"inline_rules": _HTML_INLINE_RULES,
"escape_fn": _escape_html,
},
"slack_mrkdwn": {
"code_block_formatter": _slack_code_block,
"inline_code_formatter": _slack_inline_code,
"inline_rules": _SLACK_INLINE_RULES,
"escape_fn": None,
},
"discord": {
"code_block_formatter": _discord_code_block,
"inline_code_formatter": _discord_inline_code,
"inline_rules": _DISCORD_INLINE_RULES,
"escape_fn": None,
},
"markdown": {
"code_block_formatter": _md_code_block,
"inline_code_formatter": _md_inline_code,
"inline_rules": _MD_INLINE_RULES,
"escape_fn": None,
},
"plain": {
"code_block_formatter": _plain_code_block,
"inline_code_formatter": _plain_inline_code,
"inline_rules": _PLAIN_INLINE_RULES,
"escape_fn": None,
},
}
def __init__(self, format_type: str = "plain") -> None:
self._format_type = format_type
profile = self._PROFILES.get(format_type)
if profile is None:
raise ValueError(
f"Unknown format_type: {format_type!r}. "
f"Available: {list(self._PROFILES.keys())}"
)
self._profile = profile
@property
def format_type(self) -> str:
return self._format_type
def format(self, text: str) -> str:
"""Convert Markdown *text* to the target format."""
if not text:
return text
return convert_markdown(text, **self._profile)
@classmethod
def for_channel(cls, format_type: str) -> UnifiedFormatter:
"""Factory: create a formatter for the given format type."""
return cls(format_type)
@@ -0,0 +1,42 @@
"""iMessage channel implementation for EvoScientist.
Uses imsg CLI via JSON-RPC for real-time message streaming.
Requirements:
- macOS only
- imsg CLI: brew install steipete/tap/imsg
- Full Disk Access permission
- Messages.app logged into iCloud
"""
from ..channel_manager import _parse_csv, register_channel
from .channel_rpc import IMessageChannelRpc as IMessageChannel
from .channel_rpc import IMessageConfig
from .probe import ProbeResult, probe_imessage
from .targets import (
IMessageService,
IMessageTarget,
normalize_e164,
normalize_handle,
parse_target,
)
__all__ = [
"IMessageChannel",
"IMessageConfig",
"IMessageService",
"IMessageTarget",
"ProbeResult",
"normalize_e164",
"normalize_handle",
"parse_target",
"probe_imessage",
]
def create_from_config(config) -> IMessageChannel:
allowed = _parse_csv(config.imessage_allowed_senders)
return IMessageChannel(IMessageConfig(allowed_senders=allowed))
register_channel("imessage", create_from_config)
@@ -0,0 +1,416 @@
"""iMessage channel using imsg JSON-RPC.
This is an improved implementation that uses the imsg CLI
via JSON-RPC, similar to OpenClaw's approach.
"""
import asyncio
import logging
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from ..base import Channel, ChannelError, RawIncoming
from ..config import BaseChannelConfig
from .rpc_client import ImsgRpcClient, RpcNotification
from .targets import (
ChatGuidTarget,
ChatIdentifierTarget,
ChatIdTarget,
normalize_handle,
parse_target,
)
logger = logging.getLogger(__name__)
class _IMessageAllowListMiddleware:
"""Custom allow-list middleware for iMessage's rich sender filtering.
Supports chat_id/chat_guid matching, wildcard, and normalized
phone/email matching — logic that the generic AllowListMiddleware
does not cover.
"""
def __init__(self, channel: "IMessageChannelRpc"):
self._channel = channel
async def process_inbound(self, raw, context):
chat_id = raw.metadata.get("chat_id")
chat_guid = raw.metadata.get("chat_guid")
if not self._channel._is_sender_allowed(raw.sender_id, chat_id, chat_guid):
return None
return raw
@dataclass
class IMessageConfig(BaseChannelConfig):
"""Configuration for iMessage channel."""
cli_path: str = "imsg"
db_path: str | None = None
text_chunk_limit: int = 4096
service: str = "auto" # imessage, sms, or auto
region: str = "US"
class IMessageChannelRpc(Channel):
"""iMessage channel using imsg JSON-RPC.
This implementation uses the imsg CLI via JSON-RPC over stdio,
providing real-time message streaming instead of polling.
Args:
config: Channel configuration
"""
name = "imessage"
_ready_attrs = ("_client",)
def __init__(self, config: IMessageConfig | None = None):
super().__init__(config or IMessageConfig())
self._client: ImsgRpcClient | None = None
self._subscription_id: int | None = None
self._background_tasks: set[asyncio.Task] = set()
# ── Pipeline overrides ────────────────────────────────────────
def _build_inbound_middlewares(self):
"""Use iMessage-specific allow-list middleware.
iMessage doesn't need MentionGating (always sets was_mentioned=True).
"""
from ..middleware import DedupMiddleware, GroupHistoryMiddleware
middlewares = []
middlewares.append(DedupMiddleware())
middlewares.append(_IMessageAllowListMiddleware(self))
if self.capabilities.groups:
middlewares.append(GroupHistoryMiddleware())
return middlewares
# ── Incoming message handling ─────────────────────────────────
def _handle_notification(self, notification: RpcNotification) -> None:
"""Handle incoming RPC notifications."""
if notification.method == "message":
_task = asyncio.create_task(self._handle_message(notification.params))
self._background_tasks.add(_task)
_task.add_done_callback(self._background_tasks.discard)
elif notification.method == "error":
logger.error(f"imsg error: {notification.params}")
async def _handle_message(self, params: dict | None) -> None:
"""Process incoming message notification."""
if not params:
return
message = params.get("message", {})
if not message:
return
# Skip messages from self
if message.get("is_from_me"):
return
sender = message.get("sender", "").strip()
if not sender:
return
text = message.get("text", "").strip()
# Parse timestamp
timestamp = datetime.now()
if created_at := message.get("created_at"):
try:
timestamp = datetime.fromisoformat(created_at)
except ValueError:
pass
# Build metadata
metadata = {
"chat_id": message.get("chat_id"),
"chat_guid": message.get("chat_guid"),
"is_group": message.get("is_group", False),
"chat_name": message.get("chat_name"),
}
# Handle attachments if enabled
annotations: list[str] = []
media_paths: list[str] = []
_VOICE_EXTS = {".caf", ".m4a", ".aac", ".ogg", ".opus", ".mp3", ".amr"}
if self.config.include_attachments:
attachments = message.get("attachments", [])
for att in attachments:
# imsg CLI provides local file paths for attachments
file_path = att if isinstance(att, str) else att.get("path", "")
if not file_path:
annotations.append("[attachment: missing path]")
continue
att_path = Path(file_path)
is_voice = att_path.suffix.lower() in _VOICE_EXTS
media_label = "voice" if is_voice else "attachment"
if att_path.exists():
fname = att_path.name
# Check file size before copying
from ..base import MAX_ATTACHMENT_BYTES
if att_path.stat().st_size > MAX_ATTACHMENT_BYTES:
annotations.append(
f"[{media_label}: {fname} - too large "
f"({att_path.stat().st_size} bytes)]"
)
else:
local = self._media_path(f"imsg_{fname}")
try:
import shutil
shutil.copy2(str(att_path), str(local))
media_paths.append(str(local))
annotations.append(f"[{media_label}: {local}]")
except Exception as e:
logger.warning(f"Failed to copy iMessage attachment: {e}")
annotations.append(
f"[{media_label}: {fname} - copy failed]"
)
else:
annotations.append(f"[{media_label}: {file_path} - not found]")
if not text and not media_paths and not annotations:
return
is_group = message.get("is_group", False)
await self._enqueue_raw(
RawIncoming(
sender_id=sender,
chat_id=str(metadata.get("chat_id", sender)),
text=text,
media_files=media_paths,
content_annotations=annotations,
timestamp=timestamp,
message_id=str(message.get("id", "")),
metadata=metadata,
is_group=is_group,
was_mentioned=True, # iMessage has no mention concept
)
)
# ── Sender filtering ──────────────────────────────────────────
def _is_sender_allowed(
self,
sender: str,
chat_id: int | None = None,
chat_guid: str | None = None,
) -> bool:
"""Check if sender is in allowed list.
Supports:
- Wildcard "*" to allow all
- chat_id:123 to match by chat ID
- chat_guid:abc to match by chat GUID
- Normalized phone/email matching
"""
if not self.config.allowed_senders:
return True
# Wildcard allows all
if "*" in self.config.allowed_senders:
return True
sender_normalized = normalize_handle(sender)
for entry in self.config.allowed_senders:
entry = entry.strip()
if not entry:
continue
lower = entry.lower()
# Check chat_id match
if lower.startswith("chat_id:") or lower.startswith("chatid:"):
if chat_id is not None:
try:
allowed_id = int(entry.split(":", 1)[1].strip())
if allowed_id == chat_id:
return True
except ValueError:
pass
continue
# Check chat_guid match
if lower.startswith("chat_guid:") or lower.startswith("chatguid:"):
if chat_guid:
allowed_guid = entry.split(":", 1)[1].strip()
if allowed_guid == chat_guid:
return True
continue
# Normalize and compare handle
entry_normalized = normalize_handle(entry)
if entry_normalized == sender_normalized:
return True
return False
def _normalize_sender(self, sender: str) -> str:
"""Normalize a sender identifier."""
return sender if sender.startswith("chat") else normalize_handle(sender)
def add_allowed_sender(self, sender: str) -> None:
"""Add a sender to the allowed list."""
normalized = self._normalize_sender(sender)
if self.config.allowed_senders is None:
self.config.allowed_senders = set()
self.config.allowed_senders.add(normalized)
logger.info(f"Added allowed sender: {normalized}")
def remove_allowed_sender(self, sender: str) -> None:
"""Remove a sender from the allowed list."""
normalized = self._normalize_sender(sender)
if self.config.allowed_senders:
self.config.allowed_senders.discard(normalized)
logger.info(f"Removed allowed sender: {normalized}")
def clear_allowed_senders(self) -> None:
"""Clear allowed list (allow all)."""
self.config.allowed_senders = None
logger.info("Cleared allowed senders (allowing all)")
def list_allowed_senders(self) -> list[str]:
"""Get current allowed senders."""
return list(self.config.allowed_senders) if self.config.allowed_senders else []
# ── Lifecycle ─────────────────────────────────────────────────
async def start(self) -> None:
"""Initialize and start the channel."""
logger.info("Starting iMessage channel (RPC)...")
self._client = ImsgRpcClient(
cli_path=self.config.cli_path,
db_path=self.config.db_path,
on_notification=self._handle_notification,
)
try:
await self._client.start()
except Exception as e:
raise ChannelError(f"Failed to start imsg: {e}") from e
# Subscribe to message events
try:
result = await self._client.request(
"watch.subscribe",
{"attachments": self.config.include_attachments},
)
self._subscription_id = result.get("subscription")
except Exception as e:
await self._client.stop()
raise ChannelError(f"Failed to subscribe: {e}") from e
self._running = True
logger.info("iMessage channel started")
async def _cleanup(self) -> None:
if self._client and self._subscription_id:
try:
await self._client.request(
"watch.unsubscribe",
{"subscription": self._subscription_id},
)
except Exception:
pass
if self._client:
await self._client.stop()
self._client = None
logger.info("iMessage channel stopped")
# ── Send (template method overrides) ──────────────────────────
def _resolve_target(self, chat_id: str | None, metadata: dict | None) -> dict:
"""Resolve send target from metadata or chat_id string."""
meta = metadata or {}
for key in ("chat_id", "chat_guid", "chat_identifier"):
if meta.get(key):
return {key: meta[key]}
if chat_id:
try:
target = parse_target(chat_id)
if isinstance(target, ChatIdTarget):
return {"chat_id": target.chat_id}
elif isinstance(target, ChatGuidTarget):
return {"chat_guid": target.chat_guid}
elif isinstance(target, ChatIdentifierTarget):
return {"chat_identifier": target.chat_identifier}
else:
return {"to": target.to, "service": target.service.value}
except ValueError:
return {"to": chat_id}
return {}
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
"""Send a single text chunk via iMessage RPC."""
if not self._client:
raise RuntimeError("iMessage client not running")
params: dict = {
"text": formatted_text,
"service": self.config.service,
"region": self.config.region,
}
params.update(self._resolve_target(chat_id, metadata))
if reply_to:
params["reply_to"] = reply_to
await self._client.request("send", params)
# ── Retry logic (override base) ───────────────────────────────
def _format_chunk(self, text: str) -> str:
"""iMessage uses plain text; no formatting conversion needed."""
return text
def _extract_retry_after(self, exc: Exception) -> float | None:
"""iMessage-specific retry logic.
RPC errors (e.g. AppleScript failures) are generally not
retryable. Transient connection issues get a short retry.
"""
msg = str(exc).lower()
if "not found" in msg or "applescript" in msg or "permission" in msg:
return None # not retryable
if "timeout" in msg or "connection" in msg:
return 1.0
return None # default: don't retry RPC errors
async def _send_media_impl(
self,
recipient: str,
file_path: str,
caption: str = "",
metadata: dict | None = None,
) -> bool:
"""Send a media file via iMessage."""
if not self._client:
return False
params: dict = {
"file": file_path,
"service": self.config.service,
"region": self.config.region,
}
if caption:
params["text"] = caption
target = self._resolve_target(recipient, metadata)
if not target:
logger.error("Cannot send media: no recipient")
return False
params.update(target)
await self._client.request("send", params)
return True
+109
View File
@@ -0,0 +1,109 @@
"""iMessage environment probe/diagnostics.
Provides utilities to detect and verify the imsg CLI environment.
"""
import asyncio
import shutil
from dataclasses import dataclass
@dataclass
class ProbeResult:
"""Result of iMessage environment probe."""
available: bool = False
cli_path: str | None = None
cli_version: str | None = None
rpc_supported: bool = False
error: str | None = None
def find_cli(cli_path: str = "imsg") -> str | None:
"""Find the imsg CLI binary.
Args:
cli_path: Path or command name to search
Returns:
Full path to CLI or None if not found
"""
return shutil.which(cli_path)
async def get_cli_version(cli_path: str) -> str | None:
"""Get the imsg CLI version.
Args:
cli_path: Path to CLI binary
Returns:
Version string or None
"""
try:
proc = await asyncio.create_subprocess_exec(
cli_path,
"--version",
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=5.0)
return stdout.decode().strip() or None
except Exception:
return None
async def check_rpc_support(cli_path: str) -> bool:
"""Check if CLI supports RPC mode.
Args:
cli_path: Path to CLI binary
Returns:
True if RPC is supported
"""
try:
proc = await asyncio.create_subprocess_exec(
cli_path,
"rpc",
"--help",
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
await asyncio.wait_for(proc.communicate(), timeout=5.0)
return proc.returncode == 0
except Exception:
return False
async def probe_imessage(
cli_path: str = "imsg",
timeout_ms: int = 10000,
) -> ProbeResult:
"""Probe the iMessage environment.
Args:
cli_path: Path or name of imsg CLI
timeout_ms: Timeout in milliseconds
Returns:
ProbeResult with environment details
"""
result = ProbeResult()
# Find CLI
found_path = find_cli(cli_path)
if not found_path:
result.error = f"imsg CLI not found: {cli_path}"
return result
result.cli_path = found_path
result.available = True
# Get version
result.cli_version = await get_cli_version(found_path)
# Check RPC support
result.rpc_supported = await check_rpc_support(found_path)
return result
@@ -0,0 +1,246 @@
"""JSON-RPC client for imsg CLI.
Communicates with the imsg CLI via JSON-RPC over stdio,
similar to OpenClaw's approach.
"""
import asyncio
import json
import logging
from collections.abc import Callable
from dataclasses import dataclass
from typing import Any
logger = logging.getLogger(__name__)
@dataclass
class RpcError:
"""RPC error response."""
code: int | None = None
message: str | None = None
data: Any = None
@dataclass
class RpcNotification:
"""RPC notification (no id, server-initiated)."""
method: str
params: Any = None
class ImsgRpcClient:
"""JSON-RPC client for imsg CLI.
Spawns `imsg rpc` as a subprocess and communicates via stdin/stdout.
Example:
client = ImsgRpcClient(cli_path="/usr/local/bin/imsg")
await client.start()
result = await client.request("send", {"to": "+1234", "text": "Hello"})
await client.stop()
"""
def __init__(
self,
cli_path: str = "imsg",
db_path: str | None = None,
on_notification: Callable[[RpcNotification], None] | None = None,
):
self.cli_path = cli_path
self.db_path = db_path
self.on_notification = on_notification
self._process: asyncio.subprocess.Process | None = None
self._next_id = 1
self._pending: dict[int, asyncio.Future] = {}
self._reader_task: asyncio.Task | None = None
self._stderr_task: asyncio.Task | None = None
self._closed = asyncio.Event()
async def start(self) -> None:
"""Start the imsg rpc subprocess."""
if self._process is not None:
return
args = [self.cli_path, "rpc"]
if self.db_path:
args.extend(["--db", self.db_path])
self._process = await asyncio.create_subprocess_exec(
*args,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
self._reader_task = asyncio.create_task(self._read_loop())
self._stderr_task = asyncio.create_task(self._stderr_loop())
logger.info(f"Started imsg rpc (pid={self._process.pid})")
async def stop(self) -> None:
"""Stop the imsg rpc subprocess."""
if self._process is None:
return
if self._process.stdin:
self._process.stdin.close()
if self._reader_task:
self._reader_task.cancel()
try:
await self._reader_task
except asyncio.CancelledError:
pass
if self._stderr_task:
self._stderr_task.cancel()
try:
await self._stderr_task
except asyncio.CancelledError:
pass
try:
self._process.terminate()
await asyncio.wait_for(self._process.wait(), timeout=2.0)
except TimeoutError:
self._process.kill()
await self._process.wait()
self._fail_all_pending(Exception("RPC client stopped"))
self._process = None
self._closed.set()
logger.info("Stopped imsg rpc")
async def wait_closed(self) -> None:
"""Wait for the client to close."""
await self._closed.wait()
async def request(
self,
method: str,
params: dict[str, Any] | None = None,
timeout: float = 10.0,
) -> Any:
"""Send a JSON-RPC request and wait for response.
Args:
method: RPC method name
params: Method parameters
timeout: Request timeout in seconds
Returns:
The result from the RPC response
Raises:
Exception: If request fails or times out
"""
if self._process is None or self._process.stdin is None:
raise Exception("RPC client not running")
request_id = self._next_id
self._next_id += 1
payload = {
"jsonrpc": "2.0",
"id": request_id,
"method": method,
"params": params or {},
}
future: asyncio.Future = asyncio.get_event_loop().create_future()
self._pending[request_id] = future
line = json.dumps(payload) + "\n"
self._process.stdin.write(line.encode())
await self._process.stdin.drain()
try:
return await asyncio.wait_for(future, timeout=timeout)
except TimeoutError:
self._pending.pop(request_id, None)
raise Exception(f"RPC request timeout: {method}") from None
async def _read_loop(self) -> None:
"""Read and process responses from stdout."""
if self._process is None or self._process.stdout is None:
return
while True:
try:
line = await self._process.stdout.readline()
if not line:
break
self._handle_line(line.decode().strip())
except asyncio.CancelledError:
break
except Exception as e:
logger.error(f"Error reading from imsg: {e}")
break
self._fail_all_pending(Exception("RPC connection closed"))
self._closed.set()
async def _stderr_loop(self) -> None:
"""Log stderr output."""
if self._process is None or self._process.stderr is None:
return
while True:
try:
line = await self._process.stderr.readline()
if not line:
break
logger.warning(f"imsg stderr: {line.decode().strip()}")
except asyncio.CancelledError:
break
except Exception:
break
def _handle_line(self, line: str) -> None:
"""Handle a single JSON-RPC response line."""
if not line:
return
try:
data = json.loads(line)
except json.JSONDecodeError as e:
logger.error(f"Failed to parse RPC response: {e}")
return
# Check if it's a response (has id)
if "id" in data and data["id"] is not None:
request_id = data["id"]
future = self._pending.pop(request_id, None)
if future is None:
return
if data.get("error"):
error = data["error"]
msg = error.get("message", "RPC error")
future.set_exception(Exception(msg))
else:
future.set_result(data.get("result"))
return
# It's a notification
if "method" in data:
notification = RpcNotification(
method=data["method"],
params=data.get("params"),
)
if self.on_notification:
try:
self.on_notification(notification)
except Exception as e:
logger.error(f"Notification handler error: {e}")
def _fail_all_pending(self, error: Exception) -> None:
"""Fail all pending requests with an error."""
for future in self._pending.values():
if not future.done():
future.set_exception(error)
self._pending.clear()
+89
View File
@@ -0,0 +1,89 @@
"""iMessage channel server.
Standalone script to run the iMessage channel with CLI options.
Usage:
python -m EvoScientist.channels.imessage.serve [OPTIONS]
Examples:
# Allow all senders (default)
python -m EvoScientist.channels.imessage.serve
# Only allow specific senders
python -m EvoScientist.channels.imessage.serve --allow +1234567890 --allow user@example.com
# Custom imsg path
python -m EvoScientist.channels.imessage.serve --cli-path /usr/local/bin/imsg
"""
import argparse
import logging
from ...logging_config import configure_logging_from_settings
from ..bus import MessageBus
from ..standalone import run_standalone
from . import IMessageChannel, IMessageConfig
logger = logging.getLogger(__name__)
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="iMessage channel server",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--allow",
action="append",
dest="allowed_senders",
help="Allowed sender (phone/email). Can be used multiple times.",
)
parser.add_argument(
"--cli-path",
default="imsg",
help="Path to imsg CLI (default: imsg)",
)
parser.add_argument(
"--db-path",
help="Path to Messages database",
)
parser.add_argument(
"--attachments",
action="store_true",
help="Include attachments in messages",
)
parser.add_argument(
"--agent",
action="store_true",
help="Use EvoScientist agent as handler (default: echo)",
)
parser.add_argument(
"--thinking",
action="store_true",
help="Send thinking content as intermediate messages (requires --agent)",
)
return parser.parse_args()
def main():
"""Entry point."""
configure_logging_from_settings(default_level=logging.INFO)
args = parse_args()
config = IMessageConfig(
cli_path=args.cli_path,
db_path=args.db_path,
allowed_senders=list(args.allowed_senders) if args.allowed_senders else [],
include_attachments=args.attachments,
)
send_thinking = args.thinking and args.agent
bus = MessageBus()
channel = IMessageChannel(config)
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
if __name__ == "__main__":
main()
+238
View File
@@ -0,0 +1,238 @@
"""iMessage target parsing and normalization.
Provides utilities for parsing iMessage targets and normalizing
phone numbers and email addresses, similar to OpenClaw's approach.
"""
import re
from dataclasses import dataclass
from enum import Enum
class IMessageService(Enum):
"""iMessage service type."""
IMESSAGE = "imessage"
SMS = "sms"
AUTO = "auto"
@dataclass
class ChatIdTarget:
"""Target by chat ID."""
kind: str = "chat_id"
chat_id: int = 0
@dataclass
class ChatGuidTarget:
"""Target by chat GUID."""
kind: str = "chat_guid"
chat_guid: str = ""
@dataclass
class ChatIdentifierTarget:
"""Target by chat identifier."""
kind: str = "chat_identifier"
chat_identifier: str = ""
@dataclass
class HandleTarget:
"""Target by handle (phone/email)."""
kind: str = "handle"
to: str = ""
service: IMessageService = IMessageService.AUTO
IMessageTarget = ChatIdTarget | ChatGuidTarget | ChatIdentifierTarget | HandleTarget
# Prefix constants
CHAT_ID_PREFIXES = ["chat_id:", "chatid:", "chat:"]
CHAT_GUID_PREFIXES = ["chat_guid:", "chatguid:", "guid:"]
CHAT_IDENTIFIER_PREFIXES = ["chat_identifier:", "chatidentifier:", "chatident:"]
SERVICE_PREFIXES = [
("imessage:", IMessageService.IMESSAGE),
("sms:", IMessageService.SMS),
("auto:", IMessageService.AUTO),
]
def normalize_e164(phone: str) -> str | None:
"""Normalize phone number to E.164 format.
Args:
phone: Raw phone number string
Returns:
Normalized E.164 format or None if invalid
"""
# Remove all non-digit characters except leading +
cleaned = re.sub(r"[^\d+]", "", phone)
if not cleaned:
return None
# Already has + prefix
if cleaned.startswith("+"):
digits = cleaned[1:]
if len(digits) >= 10 and len(digits) <= 15:
return cleaned
return None
# US/Canada number without country code
if len(cleaned) == 10:
return f"+1{cleaned}"
# Has country code
if len(cleaned) >= 11 and len(cleaned) <= 15:
return f"+{cleaned}"
return None
def normalize_handle(raw: str) -> str:
"""Normalize an iMessage handle (phone or email).
Args:
raw: Raw handle string
Returns:
Normalized handle
"""
trimmed = raw.strip()
if not trimmed:
return ""
lowered = trimmed.lower()
# Strip service prefixes
for prefix, _ in SERVICE_PREFIXES:
if lowered.startswith(prefix):
return normalize_handle(trimmed[len(prefix) :])
# Normalize chat_id/chat_guid/chat_identifier prefixes
for prefix in CHAT_ID_PREFIXES:
if lowered.startswith(prefix):
value = trimmed[len(prefix) :].strip()
return f"chat_id:{value}"
for prefix in CHAT_GUID_PREFIXES:
if lowered.startswith(prefix):
value = trimmed[len(prefix) :].strip()
return f"chat_guid:{value}"
for prefix in CHAT_IDENTIFIER_PREFIXES:
if lowered.startswith(prefix):
value = trimmed[len(prefix) :].strip()
return f"chat_identifier:{value}"
# Email - lowercase
if "@" in trimmed:
return trimmed.lower()
# Phone number - normalize to E.164
normalized = normalize_e164(trimmed)
if normalized:
return normalized
# Fallback: remove whitespace
return re.sub(r"\s+", "", trimmed)
def parse_target(raw: str) -> IMessageTarget:
"""Parse an iMessage target string.
Supports formats:
- chat_id:123
- chat_guid:abc-def
- chat_identifier:iMessage;+;chat123
- imessage:+1234567890
- sms:+1234567890
- +1234567890 (auto service)
- email@example.com (auto service)
Args:
raw: Raw target string
Returns:
Parsed IMessageTarget
Raises:
ValueError: If target is invalid
"""
trimmed = raw.strip()
if not trimmed:
raise ValueError("iMessage target is required")
lower = trimmed.lower()
# Check service prefixes first
for prefix, service in SERVICE_PREFIXES:
if lower.startswith(prefix):
remainder = trimmed[len(prefix) :].strip()
if not remainder:
raise ValueError(f"{prefix} target is required")
remainder_lower = remainder.lower()
# Check if remainder is a chat target
is_chat = any(
remainder_lower.startswith(p)
for p in CHAT_ID_PREFIXES
+ CHAT_GUID_PREFIXES
+ CHAT_IDENTIFIER_PREFIXES
)
if is_chat:
return parse_target(remainder)
return HandleTarget(to=remainder, service=service)
# Check chat_id prefixes
for prefix in CHAT_ID_PREFIXES:
if lower.startswith(prefix):
value = trimmed[len(prefix) :].strip()
try:
chat_id = int(value)
return ChatIdTarget(chat_id=chat_id)
except ValueError as e:
raise ValueError(f"Invalid chat_id: {value}") from e
# Check chat_guid prefixes
for prefix in CHAT_GUID_PREFIXES:
if lower.startswith(prefix):
value = trimmed[len(prefix) :].strip()
if not value:
raise ValueError("chat_guid is required")
return ChatGuidTarget(chat_guid=value)
# Check chat_identifier prefixes
for prefix in CHAT_IDENTIFIER_PREFIXES:
if lower.startswith(prefix):
value = trimmed[len(prefix) :].strip()
if not value:
raise ValueError("chat_identifier is required")
return ChatIdentifierTarget(chat_identifier=value)
# Default: handle with auto service
return HandleTarget(to=trimmed, service=IMessageService.AUTO)
def format_chat_target(chat_id: int | None) -> str:
"""Format a chat ID as a target string.
Args:
chat_id: Chat ID number
Returns:
Formatted target string or empty string
"""
if chat_id is None:
return ""
return f"chat_id:{chat_id}"
+989
View File
@@ -0,0 +1,989 @@
"""Composable message processing middleware.
Each middleware is a standalone class that can be composed into a pipeline.
They extract logic that was previously baked into the Channel base class,
making it reusable across both legacy and plugin-based channels.
Also contains the supporting data structures (DedupCache, GroupHistoryBuffer,
TypingManager, PairingManager) that were previously in separate files.
"""
from __future__ import annotations
import asyncio
import dataclasses
import logging
import random
import time
from collections import OrderedDict, deque
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from typing import Any
from .base import RawIncoming
from .bus.events import InboundMessage, OutboundMessage
from .debug import emit_debug_event_if
_logger = logging.getLogger(__name__)
# ── Task cancellation helper ─────────────────────────────────────────
async def _cancel_task(task: asyncio.Task) -> None:
"""Cancel an asyncio task and await its completion.
Suppresses ``CancelledError`` from the cancelled *task* but re-raises
if the **current** task was itself cancelled (to avoid swallowing an
outer cancellation signal — required for correct behavior on
Python 3.12+ where ``_must_cancel`` no longer auto-re-delivers).
"""
task.cancel()
try:
await task
except asyncio.CancelledError:
current = asyncio.current_task()
if current is not None and current.cancelling() > 0:
raise
except Exception:
pass # Already logged elsewhere; prevent unhandled propagation
# ═══════════════════════════════════════════════════════════════════════
# Supporting data structures
# ═══════════════════════════════════════════════════════════════════════
# ── Dedup cache ──────────────────────────────────────────────────────
_DEDUP_MAX = 1000
_DEDUP_TRIM = 500
_DEDUP_TTL = 3600 # 1 hour
class DedupCache:
"""Bounded ordered cache with TTL for detecting duplicate message IDs.
Entries expire after *ttl_seconds* and are pruned lazily on each
lookup. When the cache exceeds *max_size* entries it is trimmed
down to *trim_to* by evicting the oldest entries. Accessed entries
are moved to the end (LRU behavior).
"""
def __init__(
self,
max_size: int = _DEDUP_MAX,
trim_to: int = _DEDUP_TRIM,
ttl_seconds: float = _DEDUP_TTL,
) -> None:
self._seen: OrderedDict[str, float] = OrderedDict()
self._max = max_size
self._trim = trim_to
self._ttl = ttl_seconds
# ── public API ──────────────────────────────────────────────────
def is_duplicate(self, msg_id: str) -> bool:
"""Return ``True`` if *msg_id* has been seen before.
First-time IDs are recorded and ``False`` is returned.
Empty / falsy IDs are never considered duplicates.
Expired entries are pruned before the check.
"""
if not msg_id:
return False
self._prune()
if msg_id in self._seen:
# LRU: refresh position and timestamp
self._seen.move_to_end(msg_id)
self._seen[msg_id] = time.monotonic()
return True
self._seen[msg_id] = time.monotonic()
if len(self._seen) > self._max:
while len(self._seen) > self._trim:
self._seen.popitem(last=False)
return False
def clear(self) -> None:
"""Remove all entries."""
self._seen.clear()
@property
def size(self) -> int:
"""Number of entries currently in the cache."""
return len(self._seen)
# ── internal ────────────────────────────────────────────────────
def _prune(self) -> None:
"""Remove entries older than *ttl_seconds*."""
cutoff = time.monotonic() - self._ttl
# OrderedDict is insertion-ordered; oldest entries are first.
while self._seen:
_key, ts = next(iter(self._seen.items()))
if ts > cutoff:
break
self._seen.popitem(last=False)
# ── Group history buffer ─────────────────────────────────────────────
@dataclass
class HistoryEntry:
sender_id: str
text: str
timestamp: float
message_id: str = ""
class GroupHistoryBuffer:
"""Per-chat circular buffer of recent messages."""
def __init__(self, max_per_chat: int = 50, max_age_seconds: int = 3600):
self._buffers: dict[str, deque[HistoryEntry]] = {}
self._max = max_per_chat
self._max_age = max_age_seconds
def add(self, chat_id: str, entry: HistoryEntry) -> None:
"""Add a message to the chat's history buffer."""
if chat_id not in self._buffers:
self._buffers[chat_id] = deque(maxlen=self._max)
self._buffers[chat_id].append(entry)
def get_recent(self, chat_id: str, limit: int = 20) -> list[HistoryEntry]:
"""Get recent messages for context injection, excluding expired ones."""
buf = self._buffers.get(chat_id)
if not buf:
return []
now = time.monotonic()
recent = [e for e in buf if now - e.timestamp < self._max_age]
return recent[-limit:]
def format_context(self, chat_id: str, limit: int = 20) -> str:
"""Format recent messages as context block for the agent."""
entries = self.get_recent(chat_id, limit)
if not entries:
return ""
lines = ["[Chat messages since your last reply - for context]"]
for e in entries:
lines.append(f"[from: {e.sender_id}] {e.text}")
lines.append("[/Chat context]")
return "\n".join(lines)
def clear(self, chat_id: str) -> None:
"""Clear history for a chat (e.g., after the bot replies)."""
self._buffers.pop(chat_id, None)
# ── Typing indicator manager ─────────────────────────────────────────
class TypingManager:
"""Manages background typing-indicator loops per chat_id.
Args:
send_action: Async callable that sends a single typing indicator
for a given chat_id.
interval: Seconds between typing indicator sends.
"""
def __init__(
self,
send_action: Callable[[str], Awaitable[None]],
interval: float = 5.0,
debug_trace: bool = False,
channel_name: str = "unknown",
) -> None:
self._send_action = send_action
self._interval = interval
self._tasks: dict[str, asyncio.Task] = {}
self._debug_trace = debug_trace
self._channel_name = channel_name
async def start(self, chat_id: str) -> None:
"""Start a background typing-indicator loop for *chat_id*."""
await self.stop(chat_id)
async def _loop() -> None:
while True:
try:
await self._send_action(chat_id)
except Exception as exc:
_trace_named_event(
"typing_error",
enabled=self._debug_trace,
channel_name=self._channel_name,
chat_id=chat_id,
error=str(exc),
)
await asyncio.sleep(self._interval)
self._tasks[chat_id] = asyncio.create_task(_loop())
async def stop(self, chat_id: str) -> None:
"""Cancel the typing-indicator loop for *chat_id*."""
task = self._tasks.pop(chat_id, None)
if task:
await _cancel_task(task)
async def stop_all(self) -> None:
"""Cancel all active typing-indicator loops."""
for cid in list(self._tasks):
await self.stop(cid)
@property
def active_chats(self) -> list[str]:
"""Return chat_ids with active typing loops."""
return list(self._tasks)
# ── Pairing manager ─────────────────────────────────────────────────
@dataclass
class PairingRequest:
sender_id: str
channel: str
code: str
created_at: float
approved: bool = False
class PairingManager:
"""Manages DM pairing codes for channel access control."""
CODE_EXPIRY = 3600 # 1 hour
MAX_PENDING = 50 # max pending requests
def __init__(self):
self._pending: dict[str, PairingRequest] = {} # code -> request
self._approved: set[str] = set() # "channel:sender_id" keys
def is_approved(self, channel: str, sender_id: str) -> bool:
"""Check if sender is already approved."""
return f"{channel}:{sender_id}" in self._approved
def request_pairing(self, channel: str, sender_id: str) -> str:
"""Generate a pairing code for a new sender. Returns the code."""
# Check if already has pending request
for code, req in list(self._pending.items()):
if req.sender_id == sender_id and req.channel == channel:
if time.monotonic() - req.created_at < self.CODE_EXPIRY:
return code # return existing code
else:
del self._pending[code]
break
# Cleanup expired
self._cleanup_expired()
# Generate new code
code = f"{random.randint(100000, 999999)}"
while code in self._pending:
code = f"{random.randint(100000, 999999)}"
self._pending[code] = PairingRequest(
sender_id=sender_id,
channel=channel,
code=code,
created_at=time.monotonic(),
)
_logger.info(f"Pairing code {code} generated for {channel}:{sender_id}")
return code
def approve(self, code: str) -> tuple[bool, str]:
"""Approve a pairing code. Returns (success, message)."""
req = self._pending.get(code)
if not req:
return False, f"Unknown code: {code}"
if time.monotonic() - req.created_at > self.CODE_EXPIRY:
del self._pending[code]
return False, f"Code {code} expired"
key = f"{req.channel}:{req.sender_id}"
self._approved.add(key)
del self._pending[code]
_logger.info(f"Approved pairing for {key}")
return True, f"Approved {req.sender_id} on {req.channel}"
def reject(self, code: str) -> tuple[bool, str]:
"""Reject a pairing code."""
if code in self._pending:
del self._pending[code]
return True, f"Rejected code {code}"
return False, f"Unknown code: {code}"
def list_pending(self) -> list[PairingRequest]:
"""List all pending (non-expired) requests."""
self._cleanup_expired()
return list(self._pending.values())
def _cleanup_expired(self):
now = time.monotonic()
expired = [
c for c, r in self._pending.items() if now - r.created_at > self.CODE_EXPIRY
]
for c in expired:
del self._pending[c]
# ═══════════════════════════════════════════════════════════════════════
# Middleware classes
# ═══════════════════════════════════════════════════════════════════════
# ── Inbound middleware base ──────────────────────────────────────────
class InboundMiddleware:
"""Base class for inbound message processing middleware."""
async def process_inbound(
self,
raw: RawIncoming,
context: dict[str, Any],
) -> RawIncoming | None:
"""Process an inbound raw message.
Return the (possibly modified) RawIncoming to continue the
pipeline, or ``None`` to drop the message.
"""
return raw
class OutboundMiddlewareBase:
"""Base class for outbound message processing middleware."""
async def process_outbound(
self,
message: OutboundMessage,
context: dict[str, Any],
) -> OutboundMessage | None:
"""Process an outbound message.
Return the (possibly modified) OutboundMessage to continue,
or ``None`` to drop it.
"""
return message
def _debug_trace_enabled(context: dict[str, Any]) -> bool:
"""Check whether channel-level debug tracing is enabled for this message."""
channel = context.get("channel")
if channel is None:
return False
return channel.is_debug_trace_enabled()
def _ctx_channel_name(context: dict[str, Any]) -> str:
"""Extract the channel name from middleware context."""
ch = context.get("channel")
return getattr(ch, "name", "unknown") if ch else "unknown"
def _trace_context_event(
context: dict[str, Any],
event: str,
**fields: Any,
) -> None:
"""Emit a middleware trace event using the shared channel context."""
emit_debug_event_if(
_logger,
event,
_debug_trace_enabled(context),
channel=_ctx_channel_name(context),
**fields,
)
def _trace_named_event(
event: str,
*,
enabled: bool,
channel_name: str,
**fields: Any,
) -> None:
"""Emit a trace event for helpers that already carry trace state."""
emit_debug_event_if(
_logger,
event,
enabled,
channel=channel_name,
**fields,
)
# ── Dedup ────────────────────────────────────────────────────────────
class DedupMiddleware(InboundMiddleware):
"""Message deduplication using a bounded TTL cache."""
def __init__(
self,
max_size: int = 1000,
trim_to: int = 500,
ttl_seconds: float = 3600.0,
) -> None:
self._cache = DedupCache(
max_size=max_size,
trim_to=trim_to,
ttl_seconds=ttl_seconds,
)
async def process_inbound(
self,
raw: RawIncoming,
context: dict[str, Any],
) -> RawIncoming | None:
if raw.message_id and self._cache.is_duplicate(raw.message_id):
_trace_context_event(
context,
"middleware_dedup_drop",
message_id=raw.message_id,
sender_id=raw.sender_id,
)
return None
return raw
# ── Debounce ─────────────────────────────────────────────────────────
class DebounceMiddleware:
"""Per-sender message batching with configurable timing.
This middleware collects messages from the same sender and merges
them after a debounce delay. It does not follow the simple
process_inbound pattern because it needs to buffer across calls.
Usage: call ``submit()`` for each message; merged results are
delivered via the ``on_ready`` callback.
"""
def __init__(
self,
*,
initial_debounce: float = 2.0,
debounce_step: float = 0.5,
max_debounce: float = 5.0,
on_ready: Callable[[InboundMessage], Any] | None = None,
) -> None:
self.initial_debounce = initial_debounce
self.debounce_step = debounce_step
self.max_debounce = max_debounce
self.on_ready = on_ready
self._buffers: dict[str, list[str]] = {}
self._metadata: dict[str, dict] = {}
self._media: dict[str, list[str]] = {}
self._message_ids: dict[str, str] = {}
self._tasks: dict[str, asyncio.Task] = {}
self._channel_name: str = ""
def set_channel_name(self, name: str) -> None:
self._channel_name = name
async def submit(self, msg: InboundMessage) -> None:
"""Buffer *msg* and schedule flush after debounce delay."""
sender = msg.sender_id
if sender not in self._buffers:
self._buffers[sender] = []
self._metadata[sender] = msg.metadata
self._media[sender] = []
self._buffers[sender].append(msg.content)
if msg.message_id:
self._message_ids[sender] = msg.message_id
if msg.media:
self._media[sender].extend(msg.media)
if sender in self._tasks:
self._tasks[sender].cancel()
count = len(self._buffers[sender])
wait = min(
self.initial_debounce + (count - 1) * self.debounce_step,
self.max_debounce,
)
async def _flush(_s: str = sender, _w: float = wait) -> None:
await asyncio.sleep(_w)
await self._flush_sender(_s)
self._tasks[sender] = asyncio.create_task(_flush())
async def _flush_sender(self, sender: str) -> None:
messages = self._buffers.pop(sender, [])
metadata = self._metadata.pop(sender, None)
media = self._media.pop(sender, [])
message_id = self._message_ids.pop(sender, "")
self._tasks.pop(sender, None)
if not messages:
return
merged = "\n".join(messages)
chat_id = (metadata or {}).get("chat_id", sender)
inbound = InboundMessage(
channel=self._channel_name,
sender_id=sender,
chat_id=str(chat_id),
content=merged,
media=media,
metadata=metadata or {},
message_id=message_id,
)
if self.on_ready:
await self.on_ready(inbound)
async def cancel_all(self) -> None:
"""Cancel all pending debounce tasks and await their completion."""
tasks = list(self._tasks.values())
self._tasks.clear()
for task in tasks:
await _cancel_task(task)
# ── Chunking ─────────────────────────────────────────────────────────
class ChunkingMiddleware(OutboundMiddlewareBase):
"""Auto-split messages respecting format expansion.
Wraps the existing ``chunking.chunk_text`` utility and the
re-splitting logic from ``Channel._prepare_chunks``.
"""
def __init__(self, capabilities: Any) -> None:
from .capabilities import ChannelCapabilities
self._capabilities: ChannelCapabilities = capabilities
def prepare_chunks(
self,
content: str,
limit: int,
format_fn: Callable[[str], str] | None = None,
) -> list[tuple[str, str]]:
"""Build ``(formatted, raw)`` pairs, re-splitting when needed.
If *format_fn* is None, formatted == raw.
"""
from .base import chunk_text
if format_fn is None:
format_fn = lambda t: t # noqa: E731
raw_chunks = chunk_text(content, limit)
pairs: list[tuple[str, str]] = []
for raw in raw_chunks:
formatted = format_fn(raw)
if len(formatted) <= limit:
pairs.append((formatted, raw))
else:
sub_limit = max(limit // 2, 500)
for sub_raw in chunk_text(raw, sub_limit):
sub_fmt = format_fn(sub_raw)
if len(sub_fmt) <= limit:
pairs.append((sub_fmt, sub_raw))
else:
pairs.append((sub_raw, sub_raw))
return pairs
# ── Formatting ───────────────────────────────────────────────────────
class FormattingMiddleware(OutboundMiddlewareBase):
"""Markdown -> channel format conversion.
Uses ``UnifiedFormatter`` configured from capabilities.
"""
def __init__(self, capabilities: Any) -> None:
from .capabilities import ChannelCapabilities
from .formatter import UnifiedFormatter
caps: ChannelCapabilities = capabilities
self._formatter = UnifiedFormatter.for_channel(caps.format_type)
def format(self, text: str) -> str:
"""Convert text to channel format."""
return self._formatter.format(text)
async def process_outbound(
self,
message: OutboundMessage,
context: dict[str, Any],
) -> OutboundMessage | None:
formatted = self._formatter.format(message.content)
return dataclasses.replace(message, content=formatted)
# ── Retry ────────────────────────────────────────────────────────────
class RetryMiddleware:
"""Exponential backoff send retry.
Wraps ``retry.retry_async`` with channel-appropriate configuration.
"""
def __init__(self, channel_name: str = "unknown") -> None:
from .retry import DEFAULT_RETRY, RETRY_PRESETS
self._config = RETRY_PRESETS.get(channel_name, DEFAULT_RETRY)
self._channel_name = channel_name
async def execute(
self,
coro_factory: Callable[[], Any],
should_retry: Callable[[Exception, int], bool] | None = None,
retry_after_s: Callable[[Exception], float | None] | None = None,
) -> Any:
"""Execute *coro_factory* with retry logic."""
from .retry import retry_async
return await retry_async(
coro_factory,
config=self._config,
should_retry=should_retry or (lambda exc, _: True),
retry_after_s=retry_after_s,
on_retry=lambda info: _logger.warning(
f"{self._channel_name} retry {info.attempt}/{info.max_attempts} "
f"in {info.delay_s:.2f}s: {info.error}"
),
label=f"{self._channel_name}.send",
)
# ── Typing ───────────────────────────────────────────────────────────
class TypingMiddleware:
"""Typing indicator management.
Wraps ``TypingManager`` for use as a standalone middleware component.
"""
def __init__(
self,
send_typing_fn: Callable[[str], Any],
interval: float = 5.0,
debug_trace: bool = False,
channel_name: str = "unknown",
) -> None:
self._manager = TypingManager(
send_typing_fn,
interval=interval,
debug_trace=debug_trace,
channel_name=channel_name,
)
async def start(self, chat_id: str) -> None:
await self._manager.start(chat_id)
async def stop(self, chat_id: str) -> None:
await self._manager.stop(chat_id)
async def stop_all(self) -> None:
await self._manager.stop_all()
# ── ACK Reaction ─────────────────────────────────────────────────────
class AckReactionMiddleware:
"""ACK emoji reaction with configurable scope.
Scope controls when reactions are sent:
- ``"all"``: react to every message
- ``"direct"``: react only in DMs
- ``"group-all"``: react in group chats (all messages)
- ``"group-mentions"``: react in groups only when mentioned
- ``"off"``: disable reactions
"""
def __init__(
self,
*,
scope: str = "all",
emoji: str = "\U0001f440",
remove_after_reply: bool = False,
send_fn: Callable[[str, str, str], Any] | None = None,
remove_fn: Callable[[str, str, str], Any] | None = None,
debug_trace: bool = False,
channel_name: str = "unknown",
) -> None:
self.scope = scope
self.emoji = emoji
self.remove_after_reply = remove_after_reply
self._send_fn = send_fn
self._remove_fn = remove_fn
self._pending: dict[str, str] = {} # chat_id -> message_id
self._debug_trace = debug_trace
self._channel_name = channel_name
def should_react(self, *, is_group: bool, was_mentioned: bool) -> bool:
if self.scope == "off":
return False
if self.scope == "all":
return True
if self.scope == "direct":
return not is_group
if self.scope == "group-all":
return is_group
if self.scope == "group-mentions":
return is_group and was_mentioned
return False
async def send_ack(self, chat_id: str, message_id: str) -> None:
if self._send_fn and message_id:
try:
await self._send_fn(chat_id, message_id, self.emoji)
if self.remove_after_reply:
self._pending[chat_id] = message_id
except Exception as exc:
_trace_named_event(
"ack_send_error",
enabled=self._debug_trace,
channel_name=self._channel_name,
chat_id=chat_id,
message_id=message_id,
error=str(exc),
)
async def remove_ack(self, chat_id: str) -> None:
message_id = self._pending.pop(chat_id, None)
if message_id and self._remove_fn:
try:
await self._remove_fn(chat_id, message_id, self.emoji)
except Exception as exc:
_trace_named_event(
"ack_remove_error",
enabled=self._debug_trace,
channel_name=self._channel_name,
chat_id=chat_id,
message_id=message_id,
error=str(exc),
)
# ── Mention Gating ───────────────────────────────────────────────────
class MentionGatingMiddleware(InboundMiddleware):
"""Filter messages based on mention policy.
Policy values:
- ``"always"``: require mention in all chats
- ``"group"``: require mention only in groups (default)
- ``"off"``: never require mention
"""
def __init__(
self,
require_mention: str = "group",
strip_fn: Callable[[str], str] | None = None,
) -> None:
self.require_mention = require_mention
self._strip_fn = strip_fn
async def process_inbound(
self,
raw: RawIncoming,
context: dict[str, Any],
) -> RawIncoming | None:
if not self._should_process(raw):
_trace_context_event(
context,
"middleware_mention_drop",
chat_id=raw.chat_id,
policy=self.require_mention,
)
return None
# Strip mentions from group messages
if raw.is_group and self._strip_fn:
raw = dataclasses.replace(raw, text=self._strip_fn(raw.text))
return raw
def _should_process(self, raw: RawIncoming) -> bool:
if self.require_mention == "off":
return True
if self.require_mention == "always":
return raw.was_mentioned
# "group" — require mention only in groups
if not raw.is_group:
return True
return raw.was_mentioned
# ── AllowList ────────────────────────────────────────────────────────
class AllowListMiddleware(InboundMiddleware):
"""Sender and channel allow-list enforcement."""
def __init__(
self,
allowed_senders: set[str] | None = None,
allowed_channels: set[str] | None = None,
dm_policy: str = "allowlist",
) -> None:
self.allowed_senders = allowed_senders
self.allowed_channels = allowed_channels
self.dm_policy = dm_policy
async def process_inbound(
self,
raw: RawIncoming,
context: dict[str, Any],
) -> RawIncoming | None:
# Channel allow-list
if self.allowed_channels and str(raw.chat_id) not in self.allowed_channels:
_trace_context_event(
context,
"middleware_allowlist_drop",
sender_id=raw.sender_id,
chat_id=raw.chat_id,
reason="chat_not_allowed",
)
return None
# Sender allow-list
if not raw.is_group and self.dm_policy == "open":
return raw # open DMs bypass sender checks
if not self._is_sender_allowed(raw.sender_id):
_trace_context_event(
context,
"middleware_allowlist_drop",
sender_id=raw.sender_id,
chat_id=raw.chat_id,
reason="sender_not_allowed",
)
return None
return raw
def _is_sender_allowed(self, sender: str) -> bool:
if not self.allowed_senders:
return True
sender_str = str(sender)
if sender_str in self.allowed_senders:
return True
if "|" in sender_str:
for part in sender_str.split("|"):
if part and part in self.allowed_senders:
return True
return False
# ── Group History ────────────────────────────────────────────────────
class GroupHistoryMiddleware(InboundMiddleware):
"""Buffer non-mentioned group messages, inject as context when mentioned."""
def __init__(
self,
max_per_chat: int = 50,
max_age_seconds: int = 3600,
) -> None:
self._buffer = GroupHistoryBuffer(
max_per_chat=max_per_chat,
max_age_seconds=max_age_seconds,
)
async def process_inbound(
self,
raw: RawIncoming,
context: dict[str, Any],
) -> RawIncoming | None:
if not raw.is_group:
return raw
# Use monotonic clock for consistent expiry calculation
ts = time.monotonic()
if not raw.was_mentioned:
self._buffer.add(
raw.chat_id,
HistoryEntry(
sender_id=raw.sender_id,
text=raw.text,
timestamp=ts,
message_id=raw.message_id,
),
)
# Don't drop here — let MentionGatingMiddleware handle that
return raw
# Mentioned: inject history context
history_context = self._buffer.format_context(raw.chat_id)
if history_context:
raw = dataclasses.replace(
raw,
text=history_context
+ "\n\n[Current message - respond to this]\n"
+ raw.text,
)
self._buffer.clear(raw.chat_id)
return raw
# ── Pairing ──────────────────────────────────────────────────────────
class PairingMiddleware(InboundMiddleware):
"""DM pairing flow management.
When dm_policy is "pairing", unapproved DM senders receive a
pairing code. Approved senders pass through normally.
"""
def __init__(
self,
channel_name: str,
send_response_fn: Callable[[str, str], Any] | None = None,
dm_policy: str = "allowlist",
) -> None:
self._manager = PairingManager()
self._channel_name = channel_name
self._send_response_fn = send_response_fn
self._dm_policy = dm_policy
self._background_tasks: set[asyncio.Task] = set()
async def process_inbound(
self,
raw: RawIncoming,
context: dict[str, Any],
) -> RawIncoming | None:
if raw.is_group:
return raw # pairing only applies to DMs
if self._dm_policy != "pairing":
return raw
if self._manager.is_approved(self._channel_name, raw.sender_id):
return raw
# Request pairing
code = self._manager.request_pairing(self._channel_name, raw.sender_id)
if self._send_response_fn:
text = f"\U0001f510 Pairing required. Your code: {code}\nThis code expires in 1 hour."
task = asyncio.create_task(self._send_response_fn(raw.chat_id, text))
# Track the task to prevent GC and handle exceptions
self._background_tasks.add(task)
task.add_done_callback(self._background_tasks.discard)
_trace_context_event(
context,
"middleware_pairing_required",
sender_id=raw.sender_id,
)
_logger.info(f"Pairing required for {raw.sender_id}, code sent")
return None
+342
View File
@@ -0,0 +1,342 @@
"""Reusable channel mixins for common architecture patterns.
Three mixins that eliminate boilerplate across channels:
- ``WebhookMixin`` — aiohttp webhook server + httpx client + token refresh
- ``WebSocketMixin`` — WS connect/reconnect/heartbeat loop
- ``PollingMixin`` — async poll loop with backoff
Each mixin works with the Channel base class. Subclasses override
a small set of abstract/hook methods to define platform-specific behavior.
"""
from __future__ import annotations
import asyncio
import json
import logging
import time
from typing import Any
logger = logging.getLogger(__name__)
# ═════════════════════════════════════════════════════════════════════
# Token refresh mixin (shared by Webhook & WebSocket channels)
# ═════════════════════════════════════════════════════════════════════
class TokenMixin:
"""Mixin for channels that need OAuth-style token management.
Subclass must implement ``_fetch_token()`` returning
``(access_token, expires_in_seconds)``.
"""
_access_token: str | None = None
_token_expires: float = 0
_http_client: Any = None # httpx.AsyncClient
async def _fetch_token(self) -> tuple[str, int]:
"""Fetch a new access token. Return (token, expires_in_seconds).
Must be implemented by the channel.
"""
raise NotImplementedError
async def _refresh_token(self) -> None:
token, expire = await self._fetch_token()
self._access_token = token
self._token_expires = time.monotonic() + expire - 300
logger.debug(
f"{getattr(self, 'name', '?')} token refreshed, expires in {expire}s"
)
async def _ensure_token(self) -> str:
if not self._access_token or time.monotonic() >= self._token_expires:
await self._refresh_token()
return self._access_token
# ═════════════════════════════════════════════════════════════════════
# Webhook + REST mixin
# ═════════════════════════════════════════════════════════════════════
class WebhookMixin:
"""Mixin for channels that use an HTTP webhook server for inbound
and REST API for outbound.
Provides:
- aiohttp web server lifecycle (start/stop)
- httpx async client lifecycle
- Route registration via ``_webhook_routes()``
Subclass must implement:
- ``_webhook_routes()`` → list of (method, path, handler)
- ``_get_webhook_port()`` → int
"""
_http_client: Any = None
_runner: Any = None
_site: Any = None
def _get_webhook_port(self) -> int:
return getattr(self.config, "webhook_port", 9000)
def _webhook_routes(self) -> list[tuple[str, str, Any]]:
"""Return [(method, path, handler), ...]. Override in subclass."""
return []
async def _start_webhook_server(self) -> None:
"""Start aiohttp webhook server + httpx client.
If ``_shared_webhook_server`` is set (by ChannelManager), the
aiohttp server is already running on the shared port — only
create the httpx outbound client.
"""
import httpx
proxy = getattr(self.config, "proxy", None) or None
self._http_client = httpx.AsyncClient(timeout=15, proxy=proxy)
# Shared webhook mode: routes already registered on shared server
if getattr(self, "_shared_webhook_server", None):
logger.info(f"{getattr(self, 'name', '?')} using shared webhook server")
return
from aiohttp import web
app = web.Application()
for method, path, handler in self._webhook_routes():
if method.upper() == "GET":
app.router.add_get(path, handler)
else:
app.router.add_post(path, handler)
self._runner = web.AppRunner(app)
await self._runner.setup()
port = self._get_webhook_port()
self._site = web.TCPSite(self._runner, "0.0.0.0", port)
await self._site.start()
logger.info(f"{getattr(self, 'name', '?')} webhook on port {port}")
async def _stop_webhook_server(self) -> None:
if self._site:
await self._site.stop()
self._site = None
if self._runner:
await self._runner.cleanup()
self._runner = None
if self._http_client:
await self._http_client.aclose()
self._http_client = None
async def _api_post(
self, url: str, body: dict, headers: dict | None = None
) -> dict:
"""POST JSON to API, return parsed response. Raises on HTTP error."""
resp = await self._http_client.post(url, json=body, headers=headers)
data = resp.json()
return data
async def _api_get(self, url: str, headers: dict | None = None) -> dict:
resp = await self._http_client.get(url, headers=headers)
return resp.json()
# ═════════════════════════════════════════════════════════════════════
# WebSocket mixin
# ═════════════════════════════════════════════════════════════════════
class WebSocketMixin:
"""Mixin for channels that receive messages via WebSocket.
Provides:
- Connect/reconnect loop with exponential backoff
- Heartbeat task management
- Message dispatch
Subclass must implement:
- ``_get_ws_url()`` → WebSocket URL to connect to
- ``_on_ws_message(data)`` → handle a parsed message dict
- ``_on_ws_connected(ws)`` → called after connection (send identify, etc.)
Optional overrides:
- ``_ws_heartbeat_interval`` → seconds between heartbeats (0 = disabled)
- ``_on_ws_heartbeat(ws)`` → send heartbeat
"""
_ws_session: Any = None
_ws_heartbeat_task: asyncio.Task | None = None
_ws_heartbeat_interval: float = 0 # 0 = no heartbeat
_ws_reconnect_delay: float = 5.0
async def _get_ws_url(self) -> str:
raise NotImplementedError
async def _on_ws_connected(self, ws) -> None:
"""Called after WebSocket connects. Send identify/auth here."""
pass
async def _on_ws_message(self, data: dict | str) -> None:
"""Handle a single WebSocket message."""
raise NotImplementedError
async def _on_ws_heartbeat(self, ws) -> None:
"""Send a heartbeat. Override if needed."""
pass
async def _ws_loop(self) -> None:
"""Main WebSocket loop with auto-reconnect."""
import os
import aiohttp
while getattr(self, "_running", False):
try:
ws_url = await self._get_ws_url()
# Resolve proxy: channel config > environment variable
proxy = getattr(getattr(self, "config", None), "proxy", None)
if not proxy:
proxy = (
os.environ.get("https_proxy")
or os.environ.get("HTTPS_PROXY")
or os.environ.get("http_proxy")
or os.environ.get("HTTP_PROXY")
or None
)
logger.debug(
f"{getattr(self, 'name', '?')} WS connecting to {ws_url[:60]}... proxy={proxy}"
)
async with aiohttp.ClientSession() as session:
async with session.ws_connect(
ws_url,
proxy=proxy,
timeout=aiohttp.ClientWSTimeout(ws_close=30),
) as ws:
logger.info(f"{getattr(self, 'name', '?')} WebSocket connected")
self._ws_session = ws
await self._on_ws_connected(ws)
# Start heartbeat if configured
if self._ws_heartbeat_interval > 0:
self._ws_heartbeat_task = asyncio.create_task(
self._ws_heartbeat_loop(ws)
)
async for msg in ws:
if msg.type == aiohttp.WSMsgType.TEXT:
try:
data = json.loads(msg.data)
except (json.JSONDecodeError, TypeError):
data = msg.data
await self._on_ws_message(data)
elif msg.type in (
aiohttp.WSMsgType.CLOSED,
aiohttp.WSMsgType.ERROR,
):
break
except asyncio.CancelledError:
break
except Exception as e:
logger.error(f"{getattr(self, 'name', '?')} WS error: {e}")
await self._ws_cleanup_heartbeat()
self._ws_session = None
if getattr(self, "_running", False):
logger.info(
f"{getattr(self, 'name', '?')} reconnecting in {self._ws_reconnect_delay}s..."
)
await asyncio.sleep(self._ws_reconnect_delay)
async def _ws_heartbeat_loop(self, ws) -> None:
while True:
try:
await self._on_ws_heartbeat(ws)
except Exception:
break
await asyncio.sleep(self._ws_heartbeat_interval)
async def _ws_cleanup_heartbeat(self) -> None:
if self._ws_heartbeat_task:
task = self._ws_heartbeat_task
self._ws_heartbeat_task = None
task.cancel()
try:
await task
except asyncio.CancelledError:
current = asyncio.current_task()
if current is not None and current.cancelling() > 0:
raise
except Exception:
pass
async def _ws_send_json(self, data: dict) -> None:
"""Send JSON to the active WebSocket."""
if self._ws_session:
await self._ws_session.send_str(json.dumps(data))
async def _stop_ws(self) -> None:
await self._ws_cleanup_heartbeat()
if self._ws_session:
await self._ws_session.close()
self._ws_session = None
# ═════════════════════════════════════════════════════════════════════
# Polling mixin
# ═════════════════════════════════════════════════════════════════════
class PollingMixin:
"""Mixin for channels that poll for new messages.
Provides:
- Poll loop with configurable interval
- Error handling + reconnect
Subclass must implement:
- ``_poll_once()`` → fetch and enqueue new messages
- ``_get_poll_interval()`` → seconds between polls
"""
_poll_task: asyncio.Task | None = None
def _get_poll_interval(self) -> float:
return getattr(self.config, "poll_interval", 30)
async def _poll_once(self) -> None:
"""Fetch new messages and enqueue them. Override in subclass."""
raise NotImplementedError
async def _start_polling(self) -> None:
self._poll_task = asyncio.create_task(self._poll_loop())
async def _poll_loop(self) -> None:
interval = self._get_poll_interval()
while getattr(self, "_running", False):
try:
await self._poll_once()
except asyncio.CancelledError:
break
except Exception as e:
logger.error(f"{getattr(self, 'name', '?')} poll error: {e}")
await asyncio.sleep(interval)
async def _stop_polling(self) -> None:
if self._poll_task:
task = self._poll_task
self._poll_task = None
task.cancel()
try:
await task
except asyncio.CancelledError:
current = asyncio.current_task()
if current is not None and current.cancelling() > 0:
raise
except Exception:
pass
+243
View File
@@ -0,0 +1,243 @@
"""Plugin-based channel interface.
A ChannelPlugin is a declarative object with optional adapter slots.
The framework inspects which slots are filled and auto-assembles
the message processing pipeline.
The ``Channel`` base class extends ``ChannelPlugin``, so all channel
implementations are automatically ChannelPlugin instances.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Protocol, runtime_checkable
from .capabilities import ChannelCapabilities
# ── Channel metadata ─────────────────────────────────────────────────
@dataclass
class ChannelMeta:
"""Channel metadata for registry and UI."""
id: str
label: str
description: str = ""
docs_path: str = ""
system_image: str = "" # icon name
# ── Adapter Protocols (slots) ────────────────────────────────────────
@runtime_checkable
class ConfigAdapter(Protocol):
"""Account configuration management."""
def list_account_ids(self, config: Any) -> list[str]: ...
def resolve_account(self, config: Any, account_id: str | None = None) -> Any: ...
def is_enabled(self, account: Any, config: Any) -> bool: ...
def is_configured(self, account: Any, config: Any) -> bool: ...
@runtime_checkable
class SecurityAdapter(Protocol):
"""DM policy and security warnings."""
def resolve_dm_policy(
self, ctx: Any
) -> str: ... # "open" | "allowlist" | "pairing"
def collect_warnings(self, ctx: Any) -> list[str]: ...
@runtime_checkable
class GroupAdapter(Protocol):
"""Per-group policy resolution."""
def resolve_require_mention(self, ctx: Any) -> bool | None: ...
def resolve_tool_policy(self, ctx: Any) -> dict[str, Any] | None: ...
def resolve_intro_hint(self, ctx: Any) -> str | None: ...
@runtime_checkable
class MentionAdapter(Protocol):
"""Bot mention detection and stripping."""
def strip_mentions(self, text: str, ctx: Any) -> str: ...
@runtime_checkable
class OutboundAdapter(Protocol):
"""Outbound message delivery."""
delivery_mode: str # "direct" | "gateway" | "hybrid"
async def send_text(self, ctx: Any) -> bool: ...
async def send_media(self, ctx: Any) -> bool: ...
@runtime_checkable
class ThreadingAdapter(Protocol):
"""Reply threading behavior."""
def resolve_reply_to_mode(self, ctx: Any) -> str: ... # "off" | "first" | "all"
@runtime_checkable
class StreamingAdapter(Protocol):
"""Edit-in-place streaming output."""
async def edit_message(self, chat_id: str, message_id: str, text: str) -> bool: ...
@runtime_checkable
class DirectoryAdapter(Protocol):
"""Contact/group directory queries."""
async def list_peers(self, ctx: Any) -> list[dict]: ...
async def list_groups(self, ctx: Any) -> list[dict]: ...
async def list_group_members(self, ctx: Any) -> list[dict]: ...
@runtime_checkable
class StatusAdapter(Protocol):
"""Health probing and status reporting."""
async def probe_account(self, ctx: Any) -> Any: ...
async def audit_account(self, ctx: Any) -> Any: ...
def collect_status_issues(self, accounts: list) -> list[dict]: ...
@runtime_checkable
class HeartbeatAdapter(Protocol):
"""Channel heartbeat / readiness checks."""
async def check_ready(self, ctx: Any) -> tuple[bool, str]: ...
@runtime_checkable
class ActionsAdapter(Protocol):
"""Message actions (react, edit, delete, poll, etc.)."""
def list_actions(self) -> list[str]: ...
async def handle_action(self, action: str, ctx: Any) -> Any: ...
@runtime_checkable
class PairingAdapter(Protocol):
"""DM pairing flow."""
id_label: str
def normalize_entry(self, entry: str) -> str: ...
async def notify_approval(self, ctx: Any) -> None: ...
@runtime_checkable
class OnboardingAdapter(Protocol):
"""Interactive setup wizard hooks."""
async def wizard_steps(self, ctx: Any) -> list[dict]: ...
async def validate_step(self, step: str, value: Any) -> str | None: ...
# ── Reload policy ────────────────────────────────────────────────────
@dataclass
class ReloadPolicy:
"""Declares which config prefixes trigger a channel reload."""
config_prefixes: list[str] = field(default_factory=list)
noop_prefixes: list[str] = field(default_factory=list)
# ── ChannelPlugin ────────────────────────────────────────────────────
class ChannelPlugin:
"""Declarative channel plugin with optional adapter slots.
Replaces the monolithic Channel base class. Each slot is optional —
the framework adapts behavior based on which are present.
Usage::
class MyPlugin(ChannelPlugin):
id = "my_channel"
meta = ChannelMeta(id="my_channel", label="My Channel")
capabilities = ChannelCapabilities(...)
def __init__(self):
self.outbound = MyOutboundAdapter()
self.config_adapter = MyConfigAdapter()
async def start(self, config, account_id=None):
...
async def stop(self, account_id=None):
...
"""
id: str = ""
meta: ChannelMeta | None = None
capabilities: ChannelCapabilities = ChannelCapabilities()
# Optional adapter slots — fill what you need
# Default: SingleAccountConfigAdapter so every plugin has multi-account
# support out of the box (returns a single "default" account).
config_adapter: ConfigAdapter | None = None
def __init_subclass__(cls, **kwargs: Any) -> None:
super().__init_subclass__(**kwargs)
def __init__(self) -> None:
# Provide default SingleAccountConfigAdapter if not overridden
if self.config_adapter is None:
from .config import SingleAccountConfigAdapter
self.config_adapter = SingleAccountConfigAdapter()
security: SecurityAdapter | None = None
groups: GroupAdapter | None = None
mentions: MentionAdapter | None = None
outbound: OutboundAdapter | None = None
threading: ThreadingAdapter | None = None
streaming: StreamingAdapter | None = None
directory: DirectoryAdapter | None = None
status: StatusAdapter | None = None
heartbeat: HeartbeatAdapter | None = None
actions: ActionsAdapter | None = None
pairing: PairingAdapter | None = None
onboarding: OnboardingAdapter | None = None
# Lifecycle
reload: ReloadPolicy | None = None
# Connection management
async def start(self, config: Any, account_id: str | None = None) -> None:
"""Start the channel (or a specific account)."""
async def stop(self, account_id: str | None = None) -> None:
"""Stop the channel (or a specific account)."""
def filled_slots(self) -> list[str]:
"""Return names of adapter slots that are not None."""
slot_names = [
"config_adapter",
"security",
"groups",
"mentions",
"outbound",
"threading",
"streaming",
"directory",
"status",
"heartbeat",
"actions",
"pairing",
"onboarding",
]
return [s for s in slot_names if getattr(self, s, None) is not None]
+28
View File
@@ -0,0 +1,28 @@
"""QQ channel for EvoScientist.
Uses the official qq-botpy SDK for WebSocket connection.
Usage in config:
channel_enabled = "qq"
qq_app_id = "your_app_id"
qq_app_secret = "your_app_secret"
"""
from ..channel_manager import _parse_csv, register_channel
from .channel import QQChannel, QQConfig
__all__ = ["QQChannel", "QQConfig"]
def create_from_config(config) -> QQChannel:
allowed = _parse_csv(config.qq_allowed_senders)
return QQChannel(
QQConfig(
app_id=config.qq_app_id,
app_secret=config.qq_app_secret,
allowed_senders=allowed,
)
)
register_channel("qq", create_from_config)
+374
View File
@@ -0,0 +1,374 @@
"""QQ channel implementation using botpy SDK."""
import asyncio
import logging
from collections import deque
from dataclasses import dataclass
from datetime import datetime
from typing import ClassVar
from ..base import Channel, ChannelError, RawIncoming
from ..capabilities import QQ as QQ_CAPS
from ..config import BaseChannelConfig
from ..formatter import UnifiedFormatter
logger = logging.getLogger(__name__)
try:
import botpy
from botpy.message import C2CMessage, GroupMessage
QQ_AVAILABLE = True
except ImportError:
QQ_AVAILABLE = False
botpy = None
C2CMessage = None
GroupMessage = None
@dataclass
class QQConfig(BaseChannelConfig):
app_id: str = ""
app_secret: str = ""
text_chunk_limit: int = 4096
def _make_bot_class(channel: "QQChannel") -> "type[botpy.Client]":
"""Create a botpy Client subclass bound to the given channel."""
intents = botpy.Intents(public_messages=True, direct_message=True)
class _Bot(botpy.Client):
def __init__(self):
super().__init__(intents=intents)
async def on_ready(self):
logger.info(f"QQ bot ready: {self.robot.name}")
async def on_c2c_message_create(self, message: "C2CMessage"):
await channel._on_msg(message, "c2c")
async def on_group_at_message_create(self, message: "GroupMessage"):
await channel._on_msg(message, "group")
return _Bot
class QQChannel(Channel):
"""QQ channel using botpy SDK."""
name = "qq"
capabilities = QQ_CAPS
_ready_attrs = ("_client", "_running")
_non_retryable_patterns = ()
_mention_pattern = r"@\S+\s*"
_mention_strip_count = 1
_markdown_fallback_exc_types: ClassVar[tuple[type[Exception], ...]] = (
TypeError,
ValueError,
)
def __init__(self, config: QQConfig):
super().__init__(config)
self._client: botpy.Client | None = None
self._bot_task: asyncio.Task | None = None
self._processed_ids: deque = deque(maxlen=1000)
self._msg_seq: dict[str, int] = {} # msg_id -> next seq number
self._msg_seq_order: deque = deque(maxlen=500)
self._msg_seq_ids: set[str] = set() # companion set for O(1) lookup
self._plain_formatter = UnifiedFormatter.for_channel("plain")
# ── Lifecycle ─────────────────────────────────────────────────
async def start(self) -> None:
if not QQ_AVAILABLE:
raise ChannelError("QQ SDK not installed. Run: pip install qq-botpy")
if not self.config.app_id or not self.config.app_secret:
raise ChannelError("QQ app_id and app_secret are required")
self._running = True
BotClass = _make_bot_class(self)
self._client = BotClass()
self._bot_task = asyncio.create_task(self._run_bot())
logger.info("QQ channel starting...")
async def _run_bot(self) -> None:
try:
await self._client.start(
appid=self.config.app_id, secret=self.config.app_secret
)
except Exception as e:
logger.error(f"QQ auth failed: {e}")
self._running = False
async def _cleanup(self) -> None:
self._running = False
if self._bot_task:
self._bot_task.cancel()
try:
await self._bot_task
except asyncio.CancelledError:
pass
self._client = None
logger.info("QQ channel stopped")
# ── Incoming ──────────────────────────────────────────────────
async def _on_msg(self, message, msg_type: str) -> None:
try:
if message.id in self._processed_ids:
return
self._processed_ids.append(message.id)
author = message.author
content = (message.content or "").strip()
if msg_type == "c2c":
sender_id = str(getattr(author, "user_openid", ""))
chat_id = sender_id
else:
sender_id = str(getattr(author, "member_openid", ""))
chat_id = str(getattr(message, "group_openid", ""))
# Handle attachments (images, files, audio, video)
annotations: list[str] = []
media_paths: list[str] = []
attachments = getattr(message, "attachments", None) or []
for att in attachments:
url = getattr(att, "url", "") or ""
filename = getattr(att, "filename", "attachment") or "attachment"
content_type = getattr(att, "content_type", "") or ""
if url:
local, ann = await self._download_attachment(
url,
f"qq_{filename}",
)
if local:
media_paths.append(local)
if ann:
annotations.append(ann)
else:
annotations.append(f"[{content_type or 'attachment'}: {filename}]")
if not content and not media_paths and not annotations:
return
await self._enqueue_raw(
RawIncoming(
sender_id=sender_id,
chat_id=chat_id,
text=content,
media_files=media_paths,
content_annotations=annotations,
timestamp=datetime.now(),
message_id=message.id,
is_group=(msg_type == "group"),
was_mentioned=True,
metadata={
"chat_id": chat_id,
"msg_type": msg_type,
"event_id": message.id,
"backend": "qq",
},
)
)
except Exception as e:
logger.error(f"Error handling QQ message: {e}")
# ── Send ──────────────────────────────────────────────────────
def _next_msg_seq(self, msg_id: str) -> int:
"""Return the next msg_seq for *msg_id* and increment the counter."""
seq = self._msg_seq.get(msg_id, 1)
self._msg_seq[msg_id] = seq + 1
if msg_id not in self._msg_seq_ids:
self._msg_seq_order.append(msg_id)
self._msg_seq_ids.add(msg_id)
if len(self._msg_seq_order) > 500:
oldest = self._msg_seq_order.popleft()
self._msg_seq_ids.discard(oldest)
self._msg_seq.pop(oldest, None)
return seq
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
if not self._client:
raise ChannelError("QQ client not initialized")
msg_type = (metadata or {}).get("msg_type", "c2c")
msg_id = (metadata or {}).get("event_id", "")
seq = self._next_msg_seq(msg_id)
try:
await self._post_markdown_message(chat_id, raw_text, msg_type, msg_id, seq)
return
except Exception as exc:
if not self._should_fallback_to_plain_text(exc):
raise
self._record_markdown_fallback(chat_id, raw_text, exc)
logger.debug("QQ markdown send failed, falling back to plain text: %s", exc)
plain_text = self._plain_formatter.format(raw_text)
await self._post_plain_message(chat_id, plain_text, msg_type, msg_id, seq)
def _should_fallback_to_plain_text(self, exc: Exception) -> bool:
"""Return True only for markdown compatibility/validation failures."""
if isinstance(exc, self._markdown_fallback_exc_types):
return True
msg = str(exc).lower()
compatibility_tokens = ("unsupported", "unexpected", "unknown", "invalid")
return (
"unexpected keyword argument" in msg
or (
"markdown" in msg
and any(token in msg for token in compatibility_tokens)
)
or (
"msg_type" in msg
and any(token in msg for token in compatibility_tokens)
)
)
def _record_markdown_fallback(
self,
chat_id: str,
raw_text: str,
exc: Exception,
) -> None:
"""Emit optional debug trace for markdown fallback without blocking send."""
trace_event = getattr(self, "_trace_event", None)
if not callable(trace_event):
return
try:
trace_event(
"outbound_format_fallback",
chat_id=chat_id,
error=str(exc),
formatted_len=len(raw_text),
raw_len=len(raw_text),
)
except Exception as trace_exc:
logger.debug("QQ fallback trace failed: %s", trace_exc)
async def _post_markdown_message(
self,
chat_id: str,
text: str,
msg_type: str,
msg_id: str,
seq: int,
) -> None:
payload = {
"msg_type": 2,
"markdown": {"content": text},
"msg_id": msg_id,
"msg_seq": seq,
}
if msg_type == "group":
await self._client.api.post_group_message(
group_openid=chat_id,
**payload,
)
else:
await self._client.api.post_c2c_message(
openid=chat_id,
**payload,
)
async def _post_plain_message(
self,
chat_id: str,
text: str,
msg_type: str,
msg_id: str,
seq: int,
) -> None:
payload = {
"msg_type": 0,
"content": text,
"msg_id": msg_id,
"msg_seq": seq,
}
if msg_type == "group":
await self._client.api.post_group_message(
group_openid=chat_id,
**payload,
)
else:
await self._client.api.post_c2c_message(
openid=chat_id,
**payload,
)
# _send_typing_action: inherited no-op (QQ Bot API has no typing indicator)
# ── Media send ────────────────────────────────────────────────
# qq-botpy file_type constants: 1=image, 2=video, 3=audio
_FILE_TYPE_MAP: ClassVar[dict[str, int]] = {
".jpg": 1,
".jpeg": 1,
".png": 1,
".gif": 1,
".webp": 1,
".bmp": 1,
".mp4": 2,
".mov": 2,
".avi": 2,
".mp3": 3,
".ogg": 3,
".m4a": 3,
".wav": 3,
".silk": 3,
}
async def _send_media_impl(
self,
recipient: str,
file_path: str,
caption: str = "",
metadata: dict | None = None,
) -> bool:
"""Send a media file through QQ Bot API.
Uses post_group_file / post_c2c_file with a URL. Local files
without a public URL are not supported — falls back to a text hint.
"""
if not self._client:
raise ChannelError("QQ client not initialized")
from pathlib import Path
chat_id = self._resolve_media_chat_id(recipient, metadata)
msg_type = (metadata or {}).get("msg_type", "c2c")
ext = Path(file_path).suffix.lower()
file_type = self._FILE_TYPE_MAP.get(ext, 1) # default to image
# qq-botpy file API requires a URL, not a local path
is_url = file_path.startswith("http://") or file_path.startswith("https://")
if not is_url:
# Fallback: send text hint for local files
name = Path(file_path).name
hint = f"[文件] {name}" + (f"\n{caption}" if caption else "")
await self._send_chunk(chat_id, hint, hint, None, metadata or {})
return True
try:
if msg_type == "group":
await self._client.api.post_group_file(
group_openid=chat_id,
file_type=file_type,
url=file_path,
srv_send_msg=True,
)
else:
await self._client.api.post_c2c_file(
openid=chat_id,
file_type=file_type,
url=file_path,
srv_send_msg=True,
)
except Exception as e:
logger.warning(f"QQ media send failed: {e}")
return False
if caption:
await self._send_chunk(chat_id, caption, caption, None, metadata or {})
return True
+37
View File
@@ -0,0 +1,37 @@
"""QQ Bot credential validation."""
import logging
logger = logging.getLogger(__name__)
QQ_TOKEN_URL = "https://bots.qq.com/app/getAppAccessToken"
async def validate_qq(
app_id: str,
app_secret: str,
) -> tuple[bool, str]:
"""Validate QQ Bot credentials by fetching an access token.
Returns:
Tuple of (is_valid, message).
"""
if not app_id or not app_secret:
return False, "app_id and app_secret are required"
try:
import httpx
except ImportError:
return False, "httpx not installed"
body = {"appId": app_id, "clientSecret": app_secret}
try:
async with httpx.AsyncClient() as client:
resp = await client.post(QQ_TOKEN_URL, json=body, timeout=10)
data = resp.json()
if data.get("access_token"):
return True, "QQ Bot credentials valid"
return False, f"Error: {data.get('message', data)}"
except Exception as e:
return False, f"Error: {e}"
+84
View File
@@ -0,0 +1,84 @@
"""QQ channel server.
Standalone script to run the QQ channel with CLI options.
Usage:
python -m EvoScientist.channels.qq.serve --app-id ID --app-secret SECRET [OPTIONS]
Examples:
# Basic usage
python -m EvoScientist.channels.qq.serve --app-id ID --app-secret SECRET
# Sandbox mode with allowed senders
python -m EvoScientist.channels.qq.serve --app-id ID --app-secret SECRET --allow user123
# With agent and thinking
python -m EvoScientist.channels.qq.serve --app-id ID --app-secret SECRET --agent --thinking
"""
import argparse
import logging
from ...logging_config import configure_logging_from_settings
from ..bus import MessageBus
from ..standalone import run_standalone
from .channel import QQChannel, QQConfig
logger = logging.getLogger(__name__)
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="QQ channel server",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--app-id",
required=True,
help="QQ bot app ID",
)
parser.add_argument(
"--app-secret",
required=True,
help="QQ bot app secret",
)
parser.add_argument(
"--allow",
action="append",
dest="allowed_senders",
help="Allowed sender (QQ user ID). Can be used multiple times.",
)
parser.add_argument(
"--agent",
action="store_true",
help="Use EvoScientist agent as handler (default: echo)",
)
parser.add_argument(
"--thinking",
action="store_true",
help="Send thinking content as intermediate messages (requires --agent)",
)
return parser.parse_args()
def main():
"""Entry point."""
configure_logging_from_settings(default_level=logging.INFO)
args = parse_args()
config = QQConfig(
app_id=args.app_id,
app_secret=args.app_secret,
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
)
send_thinking = args.thinking and args.agent
bus = MessageBus()
channel = QQChannel(config)
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
if __name__ == "__main__":
main()
+114
View File
@@ -0,0 +1,114 @@
"""Configurable exponential-backoff retry for async callables."""
from __future__ import annotations
import asyncio
import random
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from typing import TypeVar
T = TypeVar("T")
@dataclass
class RetryConfig:
"""Configuration for retry behavior."""
attempts: int = 3
min_delay_s: float = 0.3
max_delay_s: float = 30.0
jitter: float = 0.1 # ±10 % random offset
@dataclass
class RetryInfo:
"""Information passed to the *on_retry* callback."""
attempt: int
max_attempts: int
delay_s: float
error: Exception
label: str | None = None
async def retry_async(
fn: Callable[[], Awaitable[T]],
config: RetryConfig | None = None,
*,
should_retry: Callable[[Exception, int], bool] | None = None,
retry_after_s: Callable[[Exception], float | None] | None = None,
on_retry: Callable[[RetryInfo], None] | None = None,
label: str | None = None,
) -> T:
"""Execute *fn* with exponential-backoff retry.
Parameters
----------
fn:
Zero-argument async factory — called on every attempt so the
awaitable is always fresh.
"""
if config is None:
config = RetryConfig()
last_exc: Exception | None = None
for attempt in range(1, config.attempts + 1):
try:
return await fn()
except Exception as exc:
last_exc = exc
if attempt >= config.attempts:
raise
if should_retry is not None and not should_retry(exc, attempt):
raise
# Compute delay
server_delay: float | None = None
if retry_after_s is not None:
server_delay = retry_after_s(exc)
if server_delay is not None:
base_delay = max(server_delay, config.min_delay_s)
else:
base_delay = config.min_delay_s * (2 ** (attempt - 1))
# Apply jitter
jittered = base_delay * (1 + random.uniform(-config.jitter, config.jitter))
# Clamp to [min_delay_s, max_delay_s]
delay = max(config.min_delay_s, min(jittered, config.max_delay_s))
if on_retry is not None:
on_retry(
RetryInfo(
attempt=attempt,
max_attempts=config.attempts,
delay_s=delay,
error=exc,
label=label,
)
)
await asyncio.sleep(delay)
# Should never reach here, but satisfy the type checker.
assert last_exc is not None
raise last_exc
# ── Presets ──────────────────────────────────────────────────────────
TELEGRAM_RETRY = RetryConfig(attempts=3, min_delay_s=0.4, max_delay_s=30.0, jitter=0.1)
DEFAULT_RETRY = RetryConfig()
# Discord, Slack, Teams, Feishu all use the same config (attempts=3,
# min_delay_s=0.5, max_delay_s=30.0, jitter=0.1) — close enough to
# DEFAULT_RETRY that separate presets add no value. Channels that
# don't appear in RETRY_PRESETS already fall back to DEFAULT_RETRY.
RETRY_PRESETS: dict[str, RetryConfig] = {
"telegram": TELEGRAM_RETRY,
}
+29
View File
@@ -0,0 +1,29 @@
"""Signal channel for EvoScientist.
Uses signal-cli in JSON RPC mode — no public IP needed.
Usage in config:
channel_enabled = "signal"
signal_phone_number = "+1234567890"
"""
from ..channel_manager import _parse_csv, register_channel
from .channel import SignalChannel, SignalConfig
__all__ = ["SignalChannel", "SignalConfig"]
def create_from_config(config) -> SignalChannel:
allowed = _parse_csv(config.signal_allowed_senders)
return SignalChannel(
SignalConfig(
phone_number=config.signal_phone_number,
cli_path=config.signal_cli_path,
config_dir=config.signal_config_dir or None,
rpc_port=config.signal_rpc_port,
allowed_senders=allowed,
)
)
register_channel("signal", create_from_config)
+510
View File
@@ -0,0 +1,510 @@
"""Signal channel implementation using signal-cli JSON RPC."""
import asyncio
import json
import logging
import re
import subprocess
from collections import deque
from dataclasses import dataclass
from datetime import datetime
from typing import Any
from ..base import Channel, ChannelError, RawIncoming
from ..capabilities import SIGNAL as SIGNAL_CAPS
from ..config import BaseChannelConfig
logger = logging.getLogger(__name__)
@dataclass
class SignalConfig(BaseChannelConfig):
phone_number: str = ""
cli_path: str = "signal-cli"
config_dir: str | None = None
rpc_port: int = 7583
text_chunk_limit: int = 4096
class SignalChannel(Channel):
"""Signal channel using signal-cli JSON RPC."""
name = "signal"
capabilities = SIGNAL_CAPS
_non_retryable_patterns = ("unregistered", "auth")
def __init__(self, config: SignalConfig):
super().__init__(config)
self._reader: asyncio.StreamReader | None = None
self._writer: asyncio.StreamWriter | None = None
self._rpc_id = 0
self._daemon_proc = None
# Pending RPC responses: rpc_id -> Future
self._pending_rpcs: dict[int, asyncio.Future] = {}
# Cache message_id → sender for reaction targetAuthor (bounded)
self._msg_senders: dict[str, str] = {}
self._msg_senders_order: deque = deque(maxlen=200)
self._listen_task: asyncio.Task | None = None
async def start(self) -> None:
if not self.config.phone_number:
raise ChannelError("Signal phone_number is required")
# Try to start signal-cli daemon if not already running
await self._ensure_daemon()
try:
# Connect to JSON RPC socket
await self._connect()
except Exception:
# If connect fails after daemon was started, clean up the daemon
await self._cleanup()
raise
self._running = True
logger.info(f"Signal channel started (phone: {self.config.phone_number})")
# Listen for incoming messages in background task
# (start() must return so that run() can iterate receive())
self._listen_task = asyncio.create_task(self._listen_loop())
async def _cleanup(self) -> None:
if self._listen_task:
self._listen_task.cancel()
self._listen_task = None
# Cancel any pending RPC futures
for fut in self._pending_rpcs.values():
if not fut.done():
fut.cancel()
self._pending_rpcs.clear()
if self._writer:
self._writer.close()
try:
await self._writer.wait_closed()
except Exception:
pass
self._writer = None
self._reader = None
if self._daemon_proc:
self._daemon_proc.terminate()
self._daemon_proc = None
logger.info("Signal channel stopped")
async def _ensure_daemon(self) -> None:
"""Start signal-cli daemon if not already running."""
try:
_reader, writer = await asyncio.wait_for(
asyncio.open_connection("localhost", self.config.rpc_port),
timeout=2,
)
writer.close()
await writer.wait_closed()
logger.info("signal-cli daemon already running")
return
except (TimeoutError, ConnectionRefusedError, OSError):
pass
# Start daemon
cmd = [self.config.cli_path, "-u", self.config.phone_number]
if self.config.config_dir:
cmd.extend(["--config", self.config.config_dir])
cmd.extend(
[
"daemon",
"--tcp",
f"localhost:{self.config.rpc_port}",
"--no-receive-stdout",
]
)
logger.info(f"Starting signal-cli daemon: {' '.join(cmd)}")
try:
self._daemon_proc = subprocess.Popen(
cmd,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
except FileNotFoundError:
raise ChannelError(
f"signal-cli not found at '{self.config.cli_path}'. "
"Install: https://github.com/AsamK/signal-cli"
) from None
# Wait for daemon to be ready
for _ in range(30):
await asyncio.sleep(1)
try:
_reader, writer = await asyncio.open_connection(
"localhost",
self.config.rpc_port,
)
writer.close()
await writer.wait_closed()
logger.info("signal-cli daemon started")
return
except (ConnectionRefusedError, OSError):
continue
raise ChannelError("signal-cli daemon failed to start within 30s")
async def _connect(self) -> None:
"""Connect to signal-cli JSON RPC socket."""
try:
self._reader, self._writer = await asyncio.open_connection(
"localhost",
self.config.rpc_port,
)
except Exception as e:
raise ChannelError(f"Cannot connect to signal-cli: {e}") from e
async def _listen_loop(self) -> None:
"""Listen for incoming JSON RPC notifications and responses."""
while self._running and self._reader:
try:
line = await self._reader.readline()
if not line:
break
data = json.loads(line.decode())
# Dispatch RPC response if it has an 'id' matching a pending call
rpc_id = data.get("id")
if rpc_id is not None and rpc_id in self._pending_rpcs:
fut = self._pending_rpcs.pop(rpc_id)
if not fut.done():
if "error" in data:
fut.set_exception(
RuntimeError(f"signal-cli RPC error: {data['error']}")
)
else:
fut.set_result(data.get("result"))
continue
await self._handle_rpc(data)
except asyncio.CancelledError:
break
except json.JSONDecodeError:
continue
except Exception as e:
logger.error(f"Signal listen error: {e}")
# Reconnect
if self._running:
await asyncio.sleep(2)
try:
await self._connect()
except Exception:
logger.warning("Signal reconnect failed, exiting listen loop")
break
async def _handle_rpc(self, data: dict) -> None:
"""Handle a JSON RPC message from signal-cli."""
method = data.get("method", "")
if method != "receive":
return
params = data.get("params", {})
envelope = params.get("envelope", {})
source = envelope.get("source") or envelope.get("sourceUuid") or ""
source_number = envelope.get("sourceNumber") or source
source_name = envelope.get("sourceName") or ""
timestamp = envelope.get("timestamp", 0)
# Ignore messages from self
if (
source_number == self.config.phone_number
or source == self.config.phone_number
):
logger.debug("Ignoring message from self")
return
# Data message (text)
data_msg = envelope.get("dataMessage", {})
if data_msg:
text = data_msg.get("message", "")
group_info = data_msg.get("groupInfo", {})
is_group = bool(group_info)
chat_id = (
group_info.get("groupId", source_number) if is_group else source_number
)
msg_ts = data_msg.get("timestamp", timestamp)
media_paths: list[str] = []
annotations: list[str] = []
_VOICE_TYPES = {
"audio/aac",
"audio/ogg",
"audio/mp4",
"audio/mpeg",
"audio/opus",
}
attachments = data_msg.get("attachments", [])
for att in attachments:
att_size = att.get("size", 0)
att_name = att.get("filename", "attachment")
att_file = att.get("file") # signal-cli provides local path
content_type = att.get("contentType", "")
is_voice = content_type in _VOICE_TYPES or att.get("voiceNote", False)
media_label = "voice" if is_voice else "attachment"
if att_file:
from pathlib import Path as _Path
att_path = _Path(att_file)
if att_path.exists():
from ..base import MAX_ATTACHMENT_BYTES
if att_path.stat().st_size > MAX_ATTACHMENT_BYTES:
annotations.append(
f"[{media_label}: {att_name} - too large ({att_path.stat().st_size} bytes)]"
)
else:
local = self._media_path(f"signal_{att_name}")
import shutil
shutil.copy2(str(att_path), str(local))
media_paths.append(str(local))
annotations.append(f"[{media_label}: {local}]")
else:
annotations.append(
f"[{media_label}: {att_name} - file not found]"
)
elif att_size:
too_large = self._check_attachment_size(att_size, att_name)
if too_large:
annotations.append(too_large)
else:
annotations.append(f"[{media_label}: {att_name}]")
if not text and not media_paths and not annotations:
if not attachments:
return
# Had attachments but none downloaded successfully
if not annotations:
text = "[attachment]"
try:
ts = datetime.fromtimestamp(msg_ts / 1000) if msg_ts else datetime.now()
except (ValueError, TypeError, OSError):
ts = datetime.now()
was_mentioned = not is_group # DMs always pass
if is_group:
mentions = data_msg.get("mentions", [])
for m in mentions:
if (
m.get("uuid") == self.config.phone_number
or m.get("number") == self.config.phone_number
):
was_mentioned = True
break
# Cache message_id → sender for reaction targetAuthor
self._cache_msg_sender(str(msg_ts), source_number)
logger.info(
"Signal message from %s: %s",
source_number,
text[:50] if text else "[media]",
)
await self._enqueue_raw(
RawIncoming(
sender_id=source_number,
chat_id=chat_id,
text=text,
content_annotations=annotations,
media_files=media_paths,
timestamp=ts,
message_id=str(msg_ts),
is_group=is_group,
was_mentioned=was_mentioned,
metadata={
"chat_id": chat_id,
"source_name": source_name,
"sender_id": source_number,
"backend": "signal",
},
)
)
# ── Typing indicator ────────────────────────────────────────────
async def _send_typing_action(self, chat_id: str) -> None:
"""Send typing indicator via signal-cli JSON RPC."""
params: dict[str, Any] = {
"account": self.config.phone_number,
}
if self._is_group_id(chat_id):
params["groupId"] = chat_id
else:
params["recipient"] = [chat_id]
try:
await self._rpc_call("sendTyping", params)
except Exception:
pass # typing indicator is best-effort
# ── ACK reaction ─────────────────────────────────────────────
def _cache_msg_sender(self, message_id: str, sender: str) -> None:
"""Store message_id → sender mapping for reaction targetAuthor."""
if len(self._msg_senders) >= 200:
oldest = self._msg_senders_order.popleft()
self._msg_senders.pop(oldest, None)
self._msg_senders[message_id] = sender
self._msg_senders_order.append(message_id)
async def _send_ack_reaction(
self, chat_id: str, message_id: str, emoji: str = "👀"
) -> None:
"""Send an acknowledgment reaction via signal-cli sendReaction."""
target_author = self._msg_senders.get(message_id, "")
if not target_author:
return # cannot send reaction without knowing the original sender
try:
params: dict[str, Any] = {
"account": self.config.phone_number,
"emoji": emoji,
"targetAuthor": target_author,
"targetTimestamp": int(message_id),
}
if self._is_group_id(chat_id):
params["groupId"] = chat_id
else:
params["recipient"] = [chat_id]
await self._rpc_call("sendReaction", params)
except Exception as e:
logger.debug(f"Signal ack reaction failed: {e}")
async def _remove_ack_reaction(
self, chat_id: str, message_id: str, emoji: str = "👀"
) -> None:
"""Remove ACK reaction via signal-cli sendReaction --remove."""
target_author = self._msg_senders.get(message_id, "")
if not target_author:
return
try:
params: dict[str, Any] = {
"account": self.config.phone_number,
"emoji": emoji,
"targetAuthor": target_author,
"targetTimestamp": int(message_id),
"remove": True,
}
if self._is_group_id(chat_id):
params["groupId"] = chat_id
else:
params["recipient"] = [chat_id]
await self._rpc_call("sendReaction", params)
except Exception as e:
logger.debug(f"Signal remove ACK reaction failed: {e}")
# ── Send ──────────────────────────────────────────────────────
@staticmethod
def _is_group_id(chat_id: str) -> bool:
"""Return True if *chat_id* looks like a Signal group ID.
Group IDs are base64-encoded strings (e.g. ``"aB3d...=="``).
Individual recipients are either phone numbers (``"+1234..."``)
or UUIDs (``"817ab5e9-..."``) — neither of which is a group.
"""
return not chat_id.startswith("+") and "-" not in chat_id
def _is_ready(self) -> bool:
return self._writer is not None and not self._writer.is_closing()
async def _rpc_call(
self, method: str, params: dict, timeout: float = 10.0
) -> dict | None:
"""Send a JSON RPC call to signal-cli and wait for the response."""
if not self._writer:
return None
self._rpc_id += 1
rpc_id = self._rpc_id
request = {
"jsonrpc": "2.0",
"id": rpc_id,
"method": method,
"params": params,
}
# Register a Future before sending so the listen loop can resolve it
loop = asyncio.get_running_loop()
fut: asyncio.Future = loop.create_future()
self._pending_rpcs[rpc_id] = fut
line = json.dumps(request) + "\n"
self._writer.write(line.encode())
await self._writer.drain()
try:
return await asyncio.wait_for(fut, timeout=timeout)
except TimeoutError:
self._pending_rpcs.pop(rpc_id, None)
logger.warning(f"Signal RPC '{method}' timed out after {timeout}s")
return None
async def _send_chunk(
self,
chat_id,
formatted_text,
raw_text,
reply_to,
metadata,
):
# Determine if group or individual
params: dict[str, Any] = {
"message": raw_text,
"account": self.config.phone_number,
}
if self._is_group_id(chat_id):
params["groupId"] = chat_id
else:
params["recipient"] = [chat_id]
await self._rpc_call("send", params)
# ── Mention stripping ────────────────────────────────────────────
def _strip_mention(self, text: str) -> str:
"""Strip bot mention from Signal messages.
Signal mentions are embedded as special objects that reference
the phone number. The text contains a placeholder character (U+FFFC)
at the mention position.
"""
phone = self.config.phone_number
if phone:
# Remove phone number if directly mentioned as text
text = re.sub(rf"@?{re.escape(phone)}\s*", "", text).strip()
# Remove Unicode Object Replacement Character used as mention placeholder
text = text.replace("\ufffc", "").strip()
return text
# ── Media send ────────────────────────────────────────────────
async def _send_media_impl(
self,
recipient: str,
file_path: str,
caption: str = "",
metadata: dict | None = None,
) -> bool:
"""Send a media file via signal-cli JSON RPC.
Uses the "send" RPC method with the attachments parameter.
"""
chat_id = self._resolve_media_chat_id(recipient, metadata)
params: dict[str, Any] = {
"account": self.config.phone_number,
"attachments": [file_path],
}
if caption:
params["message"] = caption
if self._is_group_id(chat_id):
params["groupId"] = chat_id
else:
params["recipient"] = [chat_id]
await self._rpc_call("send", params)
return True
+43
View File
@@ -0,0 +1,43 @@
"""Signal credential validation."""
import logging
logger = logging.getLogger(__name__)
async def validate_signal(
phone_number: str,
cli_path: str = "signal-cli",
rpc_port: int = 7583,
) -> tuple[bool, str]:
"""Validate Signal setup by checking signal-cli availability.
Returns:
Tuple of (is_valid, message).
"""
import asyncio
import subprocess
if not phone_number:
return False, "phone_number is required"
# Check signal-cli binary
loop = asyncio.get_event_loop()
def _check():
try:
result = subprocess.run(
[cli_path, "--version"],
capture_output=True,
text=True,
timeout=5,
)
if result.returncode == 0:
return True, f"signal-cli {result.stdout.strip()}"
return False, "signal-cli returned error"
except FileNotFoundError:
return False, f"signal-cli not found at '{cli_path}'"
except Exception as e:
return False, f"Error: {e}"
return await loop.run_in_executor(None, _check)
+96
View File
@@ -0,0 +1,96 @@
"""Signal channel server.
Standalone script to run the Signal channel with CLI options.
Usage:
python -m EvoScientist.channels.signal.serve --phone-number NUMBER [OPTIONS]
Examples:
# Basic usage
python -m EvoScientist.channels.signal.serve --phone-number +1234567890
# With custom signal-cli path and allowed senders
python -m EvoScientist.channels.signal.serve --phone-number +1234567890 --cli-path /usr/local/bin/signal-cli --allow +9876543210
# With agent and thinking
python -m EvoScientist.channels.signal.serve --phone-number +1234567890 --agent --thinking
"""
import argparse
import logging
from ...logging_config import configure_logging_from_settings
from ..bus import MessageBus
from ..standalone import run_standalone
from .channel import SignalChannel, SignalConfig
logger = logging.getLogger(__name__)
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="Signal channel server",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--phone-number",
required=True,
help="Signal phone number (e.g. +1234567890)",
)
parser.add_argument(
"--cli-path",
default="signal-cli",
help="Path to signal-cli binary (default: signal-cli)",
)
parser.add_argument(
"--config-dir",
help="signal-cli config directory",
)
parser.add_argument(
"--rpc-port",
type=int,
default=7583,
help="signal-cli JSON RPC port (default: 7583)",
)
parser.add_argument(
"--allow",
action="append",
dest="allowed_senders",
help="Allowed sender (phone number). Can be used multiple times.",
)
parser.add_argument(
"--agent",
action="store_true",
help="Use EvoScientist agent as handler (default: echo)",
)
parser.add_argument(
"--thinking",
action="store_true",
help="Send thinking content as intermediate messages (requires --agent)",
)
return parser.parse_args()
def main():
"""Entry point."""
configure_logging_from_settings(default_level=logging.INFO)
args = parse_args()
config = SignalConfig(
phone_number=args.phone_number,
cli_path=args.cli_path,
config_dir=args.config_dir,
rpc_port=args.rpc_port,
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
)
send_thinking = args.thinking and args.agent
bus = MessageBus()
channel = SignalChannel(config)
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
if __name__ == "__main__":
main()
+22
View File
@@ -0,0 +1,22 @@
from ..channel_manager import _parse_csv, register_channel
from .channel import SlackChannel, SlackConfig
__all__ = ["SlackChannel", "SlackConfig"]
def create_from_config(config) -> SlackChannel:
allowed = _parse_csv(config.slack_allowed_senders)
channels = _parse_csv(config.slack_allowed_channels)
proxy = config.slack_proxy or None
return SlackChannel(
SlackConfig(
bot_token=config.slack_bot_token,
app_token=config.slack_app_token,
allowed_senders=allowed,
allowed_channels=channels,
proxy=proxy,
)
)
register_channel("slack", create_from_config)
+302
View File
@@ -0,0 +1,302 @@
"""Slack channel implementation using slack-sdk Socket Mode."""
import asyncio
import logging
from dataclasses import dataclass
from datetime import datetime
from ..base import Channel, ChannelError, RawIncoming
from ..capabilities import SLACK as SLACK_CAPS
from ..config import BaseChannelConfig
logger = logging.getLogger(__name__)
@dataclass
class SlackConfig(BaseChannelConfig):
bot_token: str = ""
app_token: str = ""
text_chunk_limit: int = 4096
class SlackChannel(Channel):
"""Slack channel using slack-sdk Socket Mode."""
name = "slack"
capabilities = SLACK_CAPS
_ready_attrs = ("_web_client",)
_mention_pattern = r"<@{bot_id}>\s*"
def __init__(self, config: SlackConfig):
super().__init__(config)
self._socket_client = None
self._web_client = None
self._typing_message_ts: dict[str, str] = {}
async def start(self) -> None:
if not self.config.bot_token:
raise ChannelError("Slack bot token is required")
if not self.config.app_token:
raise ChannelError(
"Slack app token is required for Socket Mode (starts with xapp-)"
)
try:
from slack_sdk.socket_mode.aiohttp import SocketModeClient
from slack_sdk.socket_mode.request import SocketModeRequest
from slack_sdk.socket_mode.response import SocketModeResponse
from slack_sdk.web.async_client import AsyncWebClient
except ImportError:
raise ChannelError(
"slack-sdk or aiohttp not installed. "
"Install with: pip install evoscientist[slack]"
) from None
self._web_client = AsyncWebClient(
token=self.config.bot_token,
proxy=self._get_proxy(),
)
# Get bot user ID for filtering own messages
try:
auth = await asyncio.wait_for(
self._web_client.auth_test(),
timeout=15,
)
self._bot_user_id = auth["user_id"]
except TimeoutError:
raise ChannelError(
"Slack auth_test timed out — check network and bot token"
) from None
except Exception as e:
raise ChannelError(f"Failed to authenticate Slack bot: {e}") from e
self._socket_client = SocketModeClient(
app_token=self.config.app_token,
web_client=self._web_client,
)
async def _event_handler(
client: SocketModeClient,
req: SocketModeRequest,
) -> None:
# Acknowledge immediately
resp = SocketModeResponse(envelope_id=req.envelope_id)
await client.send_socket_mode_response(resp)
logger.debug(f"Slack socket event: type={req.type}")
if req.type == "events_api":
event = req.payload.get("event", {})
event_type = event.get("type", "")
if event_type == "message" and "subtype" not in event:
is_dm = event.get("channel_type") == "im"
await self._on_message(
event,
is_group=not is_dm,
was_mentioned=is_dm,
)
elif event_type == "app_mention":
await self._on_message(
event,
is_group=True,
was_mentioned=True,
)
self._socket_client.socket_mode_request_listeners.append(_event_handler)
try:
await asyncio.wait_for(
self._socket_client.connect(),
timeout=30,
)
except TimeoutError:
raise ChannelError(
"Slack Socket Mode connection timed out — "
"check app token (must start with xapp-) and "
"ensure Socket Mode is enabled in your Slack app settings"
) from None
self._running = True
logger.info("Slack channel started (Socket Mode)")
async def _cleanup(self) -> None:
if self._socket_client:
await self._socket_client.close()
logger.info("Slack channel stopped")
# ── Typing indicator (override base) ────────────────────────────
async def _send_typing_action(self, chat_id: str) -> None:
"""Send typing indicator via Slack.
Slack's Web API and Socket Mode do not expose a dedicated
typing-indicator endpoint for bot tokens. We approximate
the experience by posting a short-lived status message that
is deleted once the real reply is sent (handled by
``stop_typing``). When the status post fails we silently
fall back to no indicator.
"""
if not self._web_client:
return
try:
resp = await self._web_client.chat_postMessage(
channel=chat_id,
text="\u2026", # "…" ellipsis as minimal typing hint
)
ts = resp.get("ts")
if ts:
self._typing_message_ts[chat_id] = ts
except Exception:
pass
async def stop_typing(self, chat_id: str) -> None:
"""Cancel typing loop and clean up the status message."""
# Delete the ephemeral "…" message if we posted one
ts = self._typing_message_ts.pop(chat_id, None)
if ts and self._web_client:
try:
await self._web_client.chat_delete(channel=chat_id, ts=ts)
except Exception:
pass
await super().stop_typing(chat_id)
# ── Send (template method overrides) ──────────────────────────
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
kwargs = {"channel": chat_id}
# Always route to thread if thread_ts is present in metadata,
# not just for the first chunk (reply_to is only set for chunk 0).
if metadata:
thread_ts = metadata.get("thread_ts")
if thread_ts:
kwargs["thread_ts"] = thread_ts
async def _send(text):
await self._web_client.chat_postMessage(text=text, **kwargs)
await self._send_with_format_fallback(_send, formatted_text, raw_text)
async def _send_media_impl(
self,
recipient: str,
file_path: str,
caption: str = "",
metadata: dict | None = None,
) -> bool:
"""Send a media file through Slack."""
channel_id = self._resolve_media_chat_id(recipient, metadata)
await self._web_client.files_upload_v2(
channel=channel_id,
file=file_path,
initial_comment=caption or None,
)
return True
def _get_bot_identifier(self) -> str | None:
return getattr(self, "_bot_user_id", None)
# ── ACK Reactions ───────────────────────────────────────────────
async def _send_ack_reaction(
self, chat_id: str, message_id: str, emoji: str = "eyes"
) -> None:
"""Add an emoji reaction to acknowledge receipt."""
if self._web_client and message_id:
try:
await self._web_client.reactions_add(
channel=chat_id,
timestamp=message_id,
name=emoji,
)
except Exception as e:
logger.debug(f"Slack ACK reaction failed: {e}")
async def _remove_ack_reaction(
self, chat_id: str, message_id: str, emoji: str = "eyes"
) -> None:
"""Remove the ACK reaction after replying."""
if self._web_client and message_id:
try:
await self._web_client.reactions_remove(
channel=chat_id,
timestamp=message_id,
name=emoji,
)
except Exception as e:
logger.debug(f"Slack remove ACK reaction failed: {e}")
async def _on_message(
self,
event: dict,
*,
is_group: bool = False,
was_mentioned: bool = True,
) -> None:
"""Handle an incoming Slack message event."""
user_id = event.get("user", "")
# Skip bot's own messages
if user_id == getattr(self, "_bot_user_id", None):
logger.debug("Skipping own bot message")
return
# Skip bot messages (e.g. from other bots)
if event.get("bot_id"):
logger.debug(f"Skipping bot message from bot_id={event.get('bot_id')}")
return
channel_id = event.get("channel", "")
text = event.get("text", "")
annotations: list[str] = []
media_paths: list[str] = []
# Handle file attachments
if self.config.include_attachments:
files = event.get("files", [])
for file_info in files:
file_size = file_info.get("size", 0)
filename = file_info.get("name", "unknown")
url = file_info.get("url_private_download") or file_info.get(
"url_private"
)
if url and self._web_client:
headers = {"Authorization": f"Bearer {self.config.bot_token}"}
local_path, annotation = await self._download_attachment(
url,
f"{file_info.get('id', 'unknown')}_{filename}",
headers=headers,
file_size=file_size,
)
if local_path:
media_paths.append(local_path)
if annotation:
annotations.append(annotation)
ts = event.get("ts", "")
thread_ts = event.get("thread_ts") or ts
try:
timestamp = datetime.fromtimestamp(float(ts)) if ts else datetime.now()
except (ValueError, TypeError):
timestamp = datetime.now()
await self._enqueue_raw(
RawIncoming(
sender_id=user_id,
chat_id=channel_id,
text=text,
media_files=media_paths,
content_annotations=annotations,
timestamp=timestamp,
message_id=ts,
metadata={"chat_id": channel_id, "thread_ts": thread_ts},
is_group=is_group,
was_mentioned=was_mentioned,
)
)
logger.info(
f"Slack message queued: sender={user_id}, "
f"channel={channel_id}, content={text[:50]}"
)
+48
View File
@@ -0,0 +1,48 @@
"""Slack bot token validation."""
import logging
logger = logging.getLogger(__name__)
async def validate_slack_tokens(
bot_token: str,
app_token: str | None = None,
proxy: str | None = None,
) -> tuple[bool, str]:
"""Validate Slack bot token via the auth.test API.
Optionally checks the app-level token format (must start with ``xapp-``).
Returns:
Tuple of (is_valid, message).
"""
if not bot_token:
return False, "No bot token provided"
try:
import httpx
except ImportError:
return False, "httpx not installed"
# Validate bot token via auth.test
url = "https://slack.com/api/auth.test"
headers = {"Authorization": f"Bearer {bot_token}"}
try:
async with httpx.AsyncClient(proxy=proxy) as client:
resp = await client.post(url, headers=headers, timeout=10)
data = resp.json()
if not data.get("ok"):
error = data.get("error", "unknown error")
return False, f"Invalid bot token: {error}"
bot_name = data.get("user", "unknown")
team = data.get("team", "unknown")
except Exception as e:
return False, f"Error: {e}"
# Optionally validate app token format
if app_token:
if not app_token.startswith("xapp-"):
return False, "App token must start with 'xapp-'"
return True, f"Bot: {bot_name} (team: {team})"
+96
View File
@@ -0,0 +1,96 @@
"""Slack channel server.
Standalone script to run the Slack channel with CLI options.
Usage:
python -m EvoScientist.channels.slack.serve --bot-token TOKEN --app-token TOKEN [OPTIONS]
Examples:
# Allow all senders (default)
python -m EvoScientist.channels.slack.serve --bot-token xoxb-... --app-token xapp-...
# Only allow specific senders and channels
python -m EvoScientist.channels.slack.serve --bot-token xoxb-... --app-token xapp-... --allow U123 --allow-channel C456
# With proxy, agent and thinking
python -m EvoScientist.channels.slack.serve --bot-token xoxb-... --app-token xapp-... --proxy http://proxy:8080 --agent --thinking
"""
import argparse
import logging
from ...logging_config import configure_logging_from_settings
from ..bus import MessageBus
from ..standalone import run_standalone
from .channel import SlackChannel, SlackConfig
logger = logging.getLogger(__name__)
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="Slack channel server",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--bot-token",
required=True,
help="Slack bot token (xoxb-...)",
)
parser.add_argument(
"--app-token",
required=True,
help="Slack app-level token for Socket Mode (xapp-...)",
)
parser.add_argument(
"--allow",
action="append",
dest="allowed_senders",
help="Allowed sender (Slack user ID). Can be used multiple times.",
)
parser.add_argument(
"--allow-channel",
action="append",
dest="allowed_channels",
help="Allowed channel ID. Can be used multiple times.",
)
parser.add_argument(
"--proxy",
help="HTTP proxy URL for Slack API requests",
)
parser.add_argument(
"--agent",
action="store_true",
help="Use EvoScientist agent as handler (default: echo)",
)
parser.add_argument(
"--thinking",
action="store_true",
help="Send thinking content as intermediate messages (requires --agent)",
)
return parser.parse_args()
def main():
"""Entry point."""
configure_logging_from_settings(default_level=logging.INFO)
args = parse_args()
config = SlackConfig(
bot_token=args.bot_token,
app_token=args.app_token,
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
allowed_channels=set(args.allowed_channels) if args.allowed_channels else None,
proxy=args.proxy,
)
send_thinking = args.thinking and args.agent
bus = MessageBus()
channel = SlackChannel(config)
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
if __name__ == "__main__":
main()
+183
View File
@@ -0,0 +1,183 @@
"""Shared standalone runner for channel servers.
Provides the channel-agnostic agent loop that any channel can use to
run headless — consuming inbound messages from the bus, streaming
agent events, and dispatching outbound replies.
Usage from a channel's ``main()``::
from EvoScientist.channels.standalone import run_standalone
channel = SomeChannel(config)
bus = MessageBus()
run_standalone(channel, bus, use_agent=True, send_thinking=True)
"""
import asyncio
import logging
import signal
from .base import Channel
from .bus import MessageBus
from .bus.events import OutboundMessage
from .consumer import InboundConsumer
from .debug import emit_debug_event
logger = logging.getLogger(__name__)
def _channel_trace_enabled(channel: Channel) -> bool:
"""Check if debug tracing is enabled on the channel."""
try:
return channel.is_debug_trace_enabled()
except Exception:
return False
async def _deliver_outbound(channel: Channel, msg: OutboundMessage) -> None:
"""Deliver an outbound message, including any media attachments."""
if msg.content:
sent = await channel.send(msg)
if not sent:
raise RuntimeError("send() returned False")
for media_path in msg.media:
media_ok = await channel.send_media(
recipient=msg.chat_id,
file_path=media_path,
metadata=msg.metadata,
)
if not media_ok:
raise RuntimeError(f"send_media() returned False for {media_path}")
async def standalone_outbound_dispatcher(
bus: MessageBus,
channel: Channel,
) -> None:
"""Consume outbound messages from the bus and send via channel."""
while True:
try:
msg: OutboundMessage = await asyncio.wait_for(
bus.consume_outbound(),
timeout=1.0,
)
except TimeoutError:
continue
except asyncio.CancelledError:
break
try:
await _deliver_outbound(channel, msg)
except Exception as e:
emit_debug_event(
logger,
"standalone_dispatch_error",
channel=channel.name,
enabled=_channel_trace_enabled(channel),
recipient=msg.recipient,
error=str(e),
)
logger.error(f"Error sending outbound: {e}")
async def _async_main(
channel: Channel,
bus: MessageBus,
use_agent: bool,
send_thinking: bool,
) -> None:
"""Async entry point — gather channel, dispatcher and optional consumer."""
from .channel_manager import ChannelManager
channel.set_bus(bus)
if send_thinking:
channel.send_thinking = True
# Create a lightweight manager for the consumer to use
manager = ChannelManager(bus)
manager._channels[channel.name] = channel
await manager.start_health()
tasks = [channel.run()]
dispatcher = standalone_outbound_dispatcher(bus, channel)
tasks.append(dispatcher)
consumer: InboundConsumer | None = None
if use_agent:
logger.info("Loading EvoScientist agent...")
from ..EvoScientist import create_cli_agent
agent = create_cli_agent()
logger.info("Agent loaded")
consumer = InboundConsumer(
bus=bus,
manager=manager,
agent=agent,
thread_id="",
send_thinking=send_thinking,
)
manager.register_health_provider("consumer", lambda: consumer.metrics)
tasks.append(consumer.run())
if send_thinking:
logger.info("Thinking messages enabled")
async def _graceful_shutdown() -> None:
"""Graceful shutdown: drain consumer, flush outbound, stop channel."""
logger.info("Graceful shutdown initiated...")
if consumer is not None:
await consumer.stop()
# Drain outbound queue before stopping the channel
drained = 0
while True:
try:
msg = bus.outbound.get_nowait()
except asyncio.QueueEmpty:
break
try:
await asyncio.wait_for(_deliver_outbound(channel, msg), timeout=5.0)
if msg.content or msg.media:
drained += 1
except Exception:
pass
if drained:
logger.info(f"Outbound drain: {drained} sent")
channel._running = False
await channel.stop()
await manager.stop_health()
loop = asyncio.get_event_loop()
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(
sig,
lambda s=sig: asyncio.create_task(_graceful_shutdown()),
)
await asyncio.gather(*tasks)
def run_standalone(
channel: Channel,
bus: MessageBus,
*,
use_agent: bool = False,
send_thinking: bool = False,
) -> None:
"""Synchronous entry point that spins up the standalone runner.
Parameters
----------
channel:
A fully-configured :class:`Channel` instance.
bus:
The :class:`MessageBus` shared with *channel*.
use_agent:
When ``True``, load the EvoScientist agent and process inbound
messages through it.
send_thinking:
When ``True`` **and** *use_agent* is set, forward intermediate
thinking messages to the channel.
"""
asyncio.run(_async_main(channel, bus, use_agent, send_thinking))
@@ -0,0 +1,19 @@
from ..channel_manager import _parse_csv, register_channel
from .channel import TelegramChannel, TelegramConfig
__all__ = ["TelegramChannel", "TelegramConfig"]
def create_from_config(config) -> TelegramChannel:
allowed = _parse_csv(config.telegram_allowed_senders)
proxy = config.telegram_proxy or None
return TelegramChannel(
TelegramConfig(
bot_token=config.telegram_bot_token,
allowed_senders=allowed,
proxy=proxy,
)
)
register_channel("telegram", create_from_config)
+315
View File
@@ -0,0 +1,315 @@
"""Telegram channel implementation using python-telegram-bot."""
import logging
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import ClassVar
from ..base import (
AUDIO_EXTS,
IMAGE_EXTS,
VIDEO_EXTS,
Channel,
ChannelError,
RawIncoming,
)
from ..capabilities import TELEGRAM as TELEGRAM_CAPS
from ..config import BaseChannelConfig
logger = logging.getLogger(__name__)
@dataclass
class TelegramConfig(BaseChannelConfig):
bot_token: str = ""
text_chunk_limit: int = 4096
class TelegramChannel(Channel):
"""Telegram channel using python-telegram-bot with long polling."""
name = "telegram"
capabilities = TELEGRAM_CAPS
_typing_interval: float = 4.0
_ready_attrs = ("_app",)
_non_retryable_patterns = ("parse", "can't parse")
_mention_pattern = r"(?i)@{bot_id}\s*"
def __init__(self, config: TelegramConfig):
super().__init__(config)
self._app = None
self._bot_username: str = ""
async def start(self) -> None:
if not self.config.bot_token:
raise ChannelError("Telegram bot token is required")
try:
from telegram.ext import (
ApplicationBuilder,
MessageHandler,
filters,
)
except ImportError:
raise ChannelError(
"python-telegram-bot not installed. "
"Install with: pip install evoscientist[telegram]"
) from None
builder = ApplicationBuilder().token(self.config.bot_token)
if self.config.proxy:
builder = builder.proxy(self.config.proxy).get_updates_proxy(
self.config.proxy
)
self._app = builder.build()
# Accept text and media message types
media_filter = filters.TEXT
if self.config.include_attachments:
media_filter = (
filters.TEXT
| filters.PHOTO
| filters.VOICE
| filters.AUDIO
| filters.Document.ALL
| filters.VIDEO
| filters.Sticker.ALL
| filters.LOCATION
)
self._app.add_handler(
MessageHandler(media_filter & ~filters.COMMAND, self._on_message)
)
await self._app.initialize()
# Cache bot username for @mention detection in groups
bot_info = await self._app.bot.get_me()
self._bot_username = (bot_info.username or "").lower()
await self._app.start()
await self._app.updater.start_polling(drop_pending_updates=True)
self._running = True
logger.info("Telegram channel started (polling)")
async def _cleanup(self) -> None:
if self._app:
if self._app.updater and self._app.updater.running:
await self._app.updater.stop()
await self._app.stop()
await self._app.shutdown()
logger.info("Telegram channel stopped")
# ── Typing indicator (override base) ────────────────────────────
async def _send_typing_action(self, chat_id: str) -> None:
"""Send typing action via Telegram Bot API."""
if self._app:
await self._app.bot.send_chat_action(
chat_id=int(chat_id),
action="typing",
)
# ── Send (template method overrides) ──────────────────────────
async def _send_chunk(self, chat_id, formatted_text, raw_text, reply_to, metadata):
reply_id = int(reply_to) if reply_to else None
async def _send(text):
await self._app.bot.send_message(
chat_id=int(chat_id),
text=text,
parse_mode="HTML" if text == formatted_text else None,
reply_to_message_id=reply_id,
)
await self._send_with_format_fallback(_send, formatted_text, raw_text)
_MEDIA_SENDERS: ClassVar[dict] = {
IMAGE_EXTS: ("send_photo", "photo"),
VIDEO_EXTS: ("send_video", "video"),
AUDIO_EXTS: ("send_audio", "audio"),
}
async def _send_media_impl(
self,
recipient: str,
file_path: str,
caption: str = "",
metadata: dict | None = None,
) -> bool:
"""Send a media file through Telegram."""
chat_id = int(self._resolve_media_chat_id(recipient, metadata))
cap = caption or None
ext = Path(file_path).suffix.lower()
for exts, (method, param) in self._MEDIA_SENDERS.items():
if ext in exts:
await getattr(self._app.bot, method)(
chat_id=chat_id,
caption=cap,
**{param: file_path},
)
return True
await self._app.bot.send_document(
chat_id=chat_id,
document=file_path,
caption=cap,
)
return True
def _get_bot_identifier(self) -> str | None:
return self._bot_username or None
async def _send_ack_reaction(
self, chat_id: str, message_id: str, emoji: str = "👀"
) -> None:
"""Send an acknowledgment reaction via Telegram."""
if self._app:
try:
from telegram import ReactionTypeEmoji
await self._app.bot.set_message_reaction(
chat_id=int(chat_id),
message_id=int(message_id),
reaction=[ReactionTypeEmoji(emoji)],
)
except Exception as e:
logger.debug(f"Telegram ACK reaction failed: {e}")
async def _remove_ack_reaction(
self, chat_id: str, message_id: str, emoji: str = "👀"
) -> None:
"""Remove the ack reaction by setting empty reaction list."""
if self._app:
try:
await self._app.bot.set_message_reaction(
chat_id=int(chat_id),
message_id=int(message_id),
reaction=[],
)
except Exception as e:
logger.debug(f"Telegram remove ACK reaction failed: {e}")
async def _on_message(self, update, context) -> None:
"""Handler callback for text, photos, voice, audio, documents, video."""
if not update.message:
return
message = update.message
user_id = str(message.from_user.id)
chat_id = str(message.chat_id)
# Detect group and mention status for centralized gating
is_group = message.chat.type in ("group", "supergroup")
was_mentioned = True # DM default
if is_group and self._bot_username:
text_check = (message.text or message.caption or "").lower()
was_mentioned = f"@{self._bot_username}" in text_check
content_parts: list[str] = []
media_paths: list[str] = []
# Text content
if message.text:
content_parts.append(message.text)
if message.caption:
content_parts.append(message.caption)
# Handle media files
annotations: list[str] = []
if self.config.include_attachments:
media_file = None
media_type = None
if message.photo:
media_file = message.photo[-1] # Largest size
media_type = "image"
elif message.voice:
media_file = message.voice
media_type = "voice"
elif message.audio:
media_file = message.audio
media_type = "audio"
elif message.video:
media_file = message.video
media_type = "video"
elif message.document:
media_file = message.document
media_type = "file"
elif message.sticker:
media_file = message.sticker
media_type = "sticker"
# Location is not a downloadable file — handle separately
if message.location and not media_file:
loc = message.location
annotations.append(f"[位置] ({loc.latitude}, {loc.longitude})")
if media_file and self._app:
file_size = getattr(media_file, "file_size", 0) or 0
too_large = self._check_attachment_size(file_size, media_type)
if too_large:
annotations.append(too_large)
else:
try:
file = await self._app.bot.get_file(
media_file.file_id,
)
ext = self._get_extension(
media_type,
getattr(media_file, "mime_type", None),
)
file_path = self._media_path(f"{media_file.file_id[:16]}{ext}")
await file.download_to_drive(str(file_path))
media_paths.append(str(file_path))
annotations.append(f"[{media_type}: {file_path}]")
logger.debug(f"Downloaded {media_type} to {file_path}")
except Exception as e:
logger.error(f"Failed to download media: {e}")
annotations.append(f"[{media_type}: download failed]")
text_content = "\n".join(content_parts) if content_parts else ""
await self._enqueue_raw(
RawIncoming(
sender_id=user_id,
chat_id=chat_id,
text=text_content,
media_files=media_paths,
content_annotations=annotations,
timestamp=message.date or datetime.now(),
message_id=str(message.message_id),
metadata={"chat_id": chat_id},
is_group=is_group,
was_mentioned=was_mentioned,
)
)
_MIME_TO_EXT: ClassVar[dict[str, str]] = {
"image/jpeg": ".jpg",
"image/png": ".png",
"image/gif": ".gif",
"image/webp": ".webp",
"audio/ogg": ".ogg",
"audio/mpeg": ".mp3",
"audio/mp4": ".m4a",
"video/mp4": ".mp4",
"video/quicktime": ".mov",
}
_TYPE_TO_EXT: ClassVar[dict[str, str]] = {
"image": ".jpg",
"voice": ".ogg",
"audio": ".mp3",
"video": ".mp4",
"file": "",
"sticker": ".webp",
}
@staticmethod
def _get_extension(media_type: str, mime_type: str | None) -> str:
"""Get file extension based on media type and MIME type."""
if mime_type and mime_type in TelegramChannel._MIME_TO_EXT:
return TelegramChannel._MIME_TO_EXT[mime_type]
return TelegramChannel._TYPE_TO_EXT.get(media_type, "")
+34
View File
@@ -0,0 +1,34 @@
"""Telegram bot token validation."""
import logging
logger = logging.getLogger(__name__)
async def validate_telegram_token(
token: str, proxy: str | None = None
) -> tuple[bool, str]:
"""Validate a Telegram bot token via the getMe API.
Returns:
Tuple of (is_valid, message).
"""
if not token:
return False, "No token provided"
try:
import httpx
except ImportError:
return False, "httpx not installed"
url = f"https://api.telegram.org/bot{token}/getMe"
try:
async with httpx.AsyncClient(proxy=proxy) as client:
resp = await client.get(url, timeout=10)
data = resp.json()
if data.get("ok"):
username = data["result"].get("username", "unknown")
return True, f"Bot: @{username}"
return False, "Invalid token"
except Exception as e:
return False, f"Error: {e}"
+78
View File
@@ -0,0 +1,78 @@
"""Telegram channel server.
Standalone script to run the Telegram channel with CLI options.
Usage:
python -m EvoScientist.channels.telegram.serve --bot-token TOKEN [OPTIONS]
Examples:
# Allow all senders (default)
python -m EvoScientist.channels.telegram.serve --bot-token TOKEN
# Only allow specific senders
python -m EvoScientist.channels.telegram.serve --bot-token TOKEN --allow 123456 --allow 789012
# With agent and thinking
python -m EvoScientist.channels.telegram.serve --bot-token TOKEN --agent --thinking
"""
import argparse
import logging
from ...logging_config import configure_logging_from_settings
from ..bus import MessageBus
from ..standalone import run_standalone
from .channel import TelegramChannel, TelegramConfig
logger = logging.getLogger(__name__)
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="Telegram channel server",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--bot-token",
required=True,
help="Telegram bot token from @BotFather",
)
parser.add_argument(
"--allow",
action="append",
dest="allowed_senders",
help="Allowed sender (Telegram user ID). Can be used multiple times.",
)
parser.add_argument(
"--agent",
action="store_true",
help="Use EvoScientist agent as handler (default: echo)",
)
parser.add_argument(
"--thinking",
action="store_true",
help="Send thinking content as intermediate messages (requires --agent)",
)
return parser.parse_args()
def main():
"""Entry point."""
configure_logging_from_settings(default_level=logging.INFO)
args = parse_args()
config = TelegramConfig(
bot_token=args.bot_token,
allowed_senders=set(args.allowed_senders) if args.allowed_senders else None,
)
send_thinking = args.thinking and args.agent
bus = MessageBus()
channel = TelegramChannel(config)
run_standalone(channel, bus, use_agent=args.agent, send_thinking=send_thinking)
if __name__ == "__main__":
main()
+69
View File
@@ -0,0 +1,69 @@
"""WeChat channel implementations for EvoScientist.
Supports multiple WeChat backends:
- **wecom**: 企业微信应用 (WeCom / WeChat Work) via official API
— Most stable, pure HTTP, no third-party dependencies
- **wechatmp**: 微信公众号 (WeChat Official Account) via official API
— Pure HTTP webhook, suitable for public-facing bots
Both backends use httpx (already a core dependency) and receive messages
via HTTP webhook, send replies via REST API.
Usage in config:
channel_enabled = "wechat"
wechat_backend = "wecom" # or "wechatmp"
# WeCom settings
wechat_wecom_corp_id = "..."
wechat_wecom_agent_id = "..."
wechat_wecom_secret = "..."
wechat_wecom_token = "..."
wechat_wecom_encoding_aes_key = "..."
wechat_webhook_port = 9001
# OR: Official Account settings
wechat_mp_app_id = "..."
wechat_mp_app_secret = "..."
wechat_mp_token = "..."
wechat_mp_encoding_aes_key = "..."
wechat_webhook_port = 9001
"""
from ..channel_manager import _parse_csv, register_channel
from .channel import WeChatChannel, WeChatMPConfig, WeComConfig
__all__ = ["WeChatChannel", "WeChatMPConfig", "WeComConfig"]
def create_from_config(config) -> WeChatChannel:
backend = config.wechat_backend or "wecom"
allowed = _parse_csv(config.wechat_allowed_senders)
proxy = config.wechat_proxy or None
port = int(config.wechat_webhook_port or 9001)
if backend == "wechatmp":
mp_config = WeChatMPConfig(
app_id=config.wechat_mp_app_id,
app_secret=config.wechat_mp_app_secret,
token=config.wechat_mp_token,
encoding_aes_key=config.wechat_mp_encoding_aes_key,
webhook_port=port,
allowed_senders=allowed,
proxy=proxy,
)
return WeChatChannel(mp_config, backend="wechatmp")
else:
wecom_config = WeComConfig(
corp_id=config.wechat_wecom_corp_id,
agent_id=config.wechat_wecom_agent_id,
secret=config.wechat_wecom_secret,
token=config.wechat_wecom_token,
encoding_aes_key=config.wechat_wecom_encoding_aes_key,
webhook_port=port,
allowed_senders=allowed,
proxy=proxy,
)
return WeChatChannel(wecom_config, backend="wecom")
register_channel("wechat", create_from_config)
+906
View File
@@ -0,0 +1,906 @@
"""WeChat channel implementation.
Supports two backends via a unified Channel interface:
1. **wecom** (企业微信应用): Corporate WeChat official API
- Receives messages via HTTP callback (XML + optional AES encryption)
- Sends replies via REST API (POST /cgi-bin/message/send)
- Supports text, image, file, markdown messages
- Token auto-refresh with 2-hour TTL
2. **wechatmp** (微信公众号): WeChat Official Account API
- Receives messages via HTTP callback (XML + optional AES encryption)
- Sends replies via REST API (POST /cgi-bin/message/custom/send)
- Supports text, image, news messages
Both backends use httpx (already a core dependency) and aiohttp for
webhook server — matching the Feishu channel pattern.
"""
from __future__ import annotations
import asyncio
import hashlib
import logging
import re
import time
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from aiohttp import web
from ..base import Channel, ChannelError, RawIncoming
from ..capabilities import WECHAT as WECHAT_CAPS
from ..config import BaseChannelConfig
from ..mixins import TokenMixin, WebhookMixin
logger = logging.getLogger(__name__)
# ── Markdown → plain text (fallback for WeChat text messages) ────
def _strip_markdown(text: str) -> str:
"""Strip Markdown formatting for plain-text WeChat messages."""
# Remove code blocks
text = re.sub(r"```[\s\S]*?```", lambda m: m.group(0).strip("`").strip(), text)
# Remove inline code
text = re.sub(r"`([^`]+)`", r"\1", text)
# Remove bold
text = re.sub(r"\*\*(.+?)\*\*", r"\1", text)
# Remove italic
text = re.sub(r"(?<!\w)_([^_]+?)_(?!\w)", r"\1", text)
# Remove strikethrough
text = re.sub(r"~~(.+?)~~", r"\1", text)
# Convert links
text = re.sub(r"\[([^\]]+)\]\(([^)]+)\)", r"\1(\2)", text)
# Remove heading markers
text = re.sub(r"^#{1,6}\s+", "", text, flags=re.MULTILINE)
# Convert list items
text = re.sub(r"^[\-\*]\s+", "• ", text, flags=re.MULTILINE)
return text
# ── Config dataclasses ───────────────────────────────────────────
@dataclass
class WeComConfig(BaseChannelConfig):
"""Configuration for WeCom (企业微信) backend."""
corp_id: str = ""
agent_id: str = ""
secret: str = ""
token: str = ""
encoding_aes_key: str = ""
webhook_port: int = 9001
@dataclass
class WeChatMPConfig(BaseChannelConfig):
"""Configuration for WeChat Official Account (公众号) backend."""
app_id: str = ""
app_secret: str = ""
token: str = ""
encoding_aes_key: str = ""
webhook_port: int = 9001
# ── Unified WeChat Channel ───────────────────────────────────────
class WeChatChannel(Channel, WebhookMixin, TokenMixin):
capabilities = WECHAT_CAPS
"""Unified WeChat channel supporting WeCom and Official Account backends.
Architecture follows the same pattern as FeishuChannel:
- HTTP webhook server (aiohttp) for inbound messages
- REST API calls (httpx) for outbound messages
- Token auto-refresh
"""
name = "wechat"
_typing_interval: float = 5.0 # WeChat has no typing API, but keep for interface
_ready_attrs = ("_http_client", "_access_token")
_rate_limit_patterns = ("45009", "frequency", "freq")
_rate_limit_delay = 2.0
_mention_pattern = r"@\S+\s*"
_mention_strip_count = 1
def __init__(
self,
config: WeComConfig | WeChatMPConfig,
backend: str = "wecom",
):
super().__init__(config)
self._backend = backend
self._access_token: str | None = None
self._token_expires: float = 0
self._runner = None
self._site = None
self._http_client = None
self._crypto = None # WeChatCrypto instance (optional)
self._typing_message_ids: dict[
str, list[str]
] = {} # chat_id → [msgid, ...] for typing recall
self._background_tasks: set[asyncio.Task] = set()
# ── Lifecycle ─────────────────────────────────────────────────
def _webhook_routes(self) -> list[tuple[str, str, Any]]:
"""Return HTTP routes for the shared webhook server."""
return [
("GET", "/wechat/callback", self._handle_verify),
("POST", "/wechat/callback", self._handle_message),
]
async def start(self) -> None:
try:
import httpx
from aiohttp import web
except ImportError:
raise ChannelError(
"aiohttp or httpx not installed. "
"Install with: pip install aiohttp httpx"
) from None
self._validate_config()
import httpx
self._http_client = httpx.AsyncClient(
timeout=15,
proxy=self._get_proxy(),
)
# Set up message encryption if configured
if self.config.encoding_aes_key and self.config.token:
from .crypto import WeChatCrypto
app_id = self._get_app_id()
self._crypto = WeChatCrypto(
token=self.config.token,
encoding_aes_key=self.config.encoding_aes_key,
app_id=app_id,
)
# Verify credentials by fetching initial token
await self._refresh_token()
if not getattr(self, "_shared_webhook_server", None):
app = web.Application()
app.router.add_get("/wechat/callback", self._handle_verify)
app.router.add_post("/wechat/callback", self._handle_message)
self._runner = web.AppRunner(app)
await self._runner.setup()
self._site = web.TCPSite(
self._runner,
"0.0.0.0",
self.config.webhook_port,
)
await self._site.start()
self._running = True
logger.info(
f"WeChat channel started "
f"(backend={self._backend}, "
f"webhook on port {self.config.webhook_port})"
)
async def _cleanup(self) -> None:
if self._site:
await self._site.stop()
if self._runner:
await self._runner.cleanup()
if self._http_client:
await self._http_client.aclose()
self._http_client = None
self._access_token = None
logger.info("WeChat channel stopped")
def _validate_config(self) -> None:
"""Validate required config fields based on backend."""
if self._backend == "wecom":
cfg = self.config
if not cfg.corp_id:
raise ChannelError("WeCom corp_id is required")
if not cfg.secret:
raise ChannelError("WeCom secret is required")
if not cfg.agent_id:
raise ChannelError("WeCom agent_id is required")
elif self._backend == "wechatmp":
cfg = self.config
if not cfg.app_id:
raise ChannelError("WeChat MP app_id is required")
if not cfg.app_secret:
raise ChannelError("WeChat MP app_secret is required")
def _get_app_id(self) -> str:
"""Return the app identifier for crypto operations."""
if self._backend == "wecom":
return self.config.corp_id
return self.config.app_id
# ── Token management ──────────────────────────────────────────
async def _refresh_token(self) -> None:
"""Fetch or refresh the access_token."""
if self._backend == "wecom":
url = (
f"https://qyapi.weixin.qq.com/cgi-bin/gettoken"
f"?corpid={self.config.corp_id}"
f"&corpsecret={self.config.secret}"
)
else:
url = (
f"https://api.weixin.qq.com/cgi-bin/token"
f"?grant_type=client_credential"
f"&appid={self.config.app_id}"
f"&secret={self.config.app_secret}"
)
try:
resp = await self._http_client.get(url)
data = resp.json()
except Exception as e:
if not self._running:
raise ChannelError(f"Failed to get WeChat access token: {e}") from e
raise RuntimeError(f"Failed to get WeChat access token: {e}") from e
if data.get("errcode", 0) != 0:
err_msg = (
f"WeChat auth error ({data.get('errcode')}): "
f"{data.get('errmsg', 'unknown')}"
)
if not self._running:
raise ChannelError(err_msg)
raise RuntimeError(err_msg)
self._access_token = data["access_token"]
expire = data.get("expires_in", 7200)
# Refresh 5 minutes before expiry
self._token_expires = time.monotonic() + expire - 300
logger.debug(f"WeChat token refreshed, expires in {expire}s")
async def _ensure_token(self) -> str:
"""Return a valid access token, refreshing if needed."""
if not self._access_token or time.monotonic() >= self._token_expires:
await self._refresh_token()
return self._access_token
# ── Signature verification (GET callback) ─────────────────────
async def _handle_verify(self, request) -> web.Response:
"""Handle GET /wechat/callback for URL verification.
WeChat/WeCom sends: msg_signature, timestamp, nonce, echostr
We decrypt echostr (encrypted mode) or verify signature (plain mode)
and return the plain echostr.
"""
from aiohttp import web
signature = request.query.get("msg_signature") or request.query.get(
"signature", ""
)
timestamp = request.query.get("timestamp", "")
nonce = request.query.get("nonce", "")
echostr = request.query.get("echostr", "")
logger.info(f"Verify request received: timestamp={timestamp}")
if not echostr:
return web.Response(status=400, text="missing echostr")
# Encrypted mode: WeCom sends msg_signature and encrypted echostr
if self._crypto and request.query.get("msg_signature"):
# Verify signature first
sig_ok = self._crypto.verify_signature(signature, timestamp, nonce, echostr)
if not sig_ok:
logger.warning("WeChat verify: signature mismatch")
# Try to decrypt regardless — the decrypted echostr must be returned
try:
plain_echostr, _ = self._crypto.decrypt(echostr)
logger.info("WeChat verify: echostr decrypted successfully")
return web.Response(text=plain_echostr)
except Exception as e:
logger.error(f"WeChat verify: echostr decrypt failed: {e}")
return web.Response(status=500)
else:
# Plain mode verification
token = self.config.token
if token:
parts = sorted([token, timestamp, nonce])
expected = hashlib.sha1("".join(parts).encode()).hexdigest()
if expected != signature:
logger.warning("WeChat verify: signature mismatch (plain)")
return web.Response(status=403)
return web.Response(text=echostr)
# ── Inbound message handling (POST callback) ──────────────────
async def _handle_message(self, request) -> web.Response:
"""Handle POST /wechat/callback for incoming messages."""
from aiohttp import web
from .crypto import parse_xml
try:
body = await request.text()
except Exception:
return web.Response(status=400)
logger.info(f"WeChat callback POST received, body length={len(body)}")
xml_data = parse_xml(body)
# If encrypted, decrypt first
encrypt = xml_data.get("Encrypt", "")
if encrypt and self._crypto:
signature = request.query.get("msg_signature", "")
timestamp = request.query.get("timestamp", "")
nonce = request.query.get("nonce", "")
if not self._crypto.verify_signature(signature, timestamp, nonce, encrypt):
logger.warning("WeChat message signature mismatch")
return web.Response(status=403)
try:
decrypted_xml, _from_id = self._crypto.decrypt(encrypt)
xml_data = parse_xml(decrypted_xml)
except Exception as e:
logger.error(f"WeChat decrypt failed: {e}")
return web.Response(status=500)
# Process message asynchronously — WeCom requires a response within
# 5 seconds, but media downloads can take much longer. Return
# "success" immediately and handle the message in the background.
_task = asyncio.create_task(self._safe_process_message(xml_data))
self._background_tasks.add(_task)
_task.add_done_callback(self._background_tasks.discard)
return web.Response(text="success")
async def _safe_process_message(self, xml_data: dict[str, str]) -> None:
"""Wrapper that catches exceptions so fire-and-forget tasks don't leak."""
try:
await self._process_message(xml_data)
except Exception:
logger.exception("Error processing WeChat message")
async def _process_message(self, xml_data: dict[str, str]) -> None:
"""Process a parsed XML message from WeChat/WeCom callback."""
msg_type = xml_data.get("MsgType", "")
from_user = xml_data.get("FromUserName", "")
to_user = xml_data.get("ToUserName", "")
content = xml_data.get("Content", "")
msg_id = xml_data.get("MsgId", "")
create_time = xml_data.get("CreateTime", "")
logger.info(
f"WeChat message received: type={msg_type}, from={from_user}, id={msg_id}, keys={list(xml_data.keys())}"
)
if not from_user:
return
# Determine chat_id
# For WeCom: FromUserName is the user's UserID
# For MP: FromUserName is the user's OpenID
chat_id = from_user
# Group chat detection
is_group = False
was_mentioned = True # Default: treat as mentioned (DMs)
# WeCom group detection: ChatId field indicates a group message
if self._backend == "wecom":
group_chat_id = xml_data.get("ChatId", "")
if group_chat_id:
is_group = True
chat_id = group_chat_id
# WeCom sets MsgType=event with Event=sys when bot is @mentioned,
# but for text messages we check the XML AtUserList field
at_user_list = xml_data.get("AtUserList", "")
was_mentioned = bool(at_user_list)
# Handle different message types
text = ""
annotations: list[str] = []
media_paths: list[str] = []
if msg_type == "text":
text = content
elif msg_type == "image":
pic_url = xml_data.get("PicUrl", "")
media_id = xml_data.get("MediaId", "")
if pic_url:
local, ann = await self._download_attachment(
pic_url,
f"wechat_{msg_id}.jpg",
)
if local:
media_paths.append(local)
if ann:
annotations.append(ann)
elif media_id:
local, ann = await self._download_wechat_media(
media_id,
f"wechat_image_{msg_id}",
)
if local:
media_paths.append(local)
if ann:
annotations.append(ann)
else:
annotations.append("[image: no download source]")
elif msg_type == "voice":
recognition = xml_data.get("Recognition", "")
media_id = xml_data.get("MediaId", "")
if media_id:
local, ann = await self._download_wechat_media(
media_id, f"wechat_voice_{msg_id}"
)
if local:
media_paths.append(local)
if ann:
ann = ann.replace("[attachment:", "[voice:")
annotations.append(ann)
if recognition:
text = f"[语音识别] {recognition}"
elif not media_paths:
annotations.append("[voice message]")
elif msg_type in ("video", "shortvideo"):
media_id = xml_data.get("MediaId", "")
if media_id:
local, ann = await self._download_wechat_media(
media_id, f"wechat_{msg_type}_{msg_id}"
)
if local:
media_paths.append(local)
if ann:
annotations.append(ann)
if not media_paths:
annotations.append(f"[{msg_type} message]")
elif msg_type == "location":
label = xml_data.get("Label", "")
lat = xml_data.get("Location_X", "")
lon = xml_data.get("Location_Y", "")
text = f"[位置] {label} ({lat}, {lon})"
elif msg_type == "file":
media_id = xml_data.get("MediaId", "")
file_name = xml_data.get("FileName", "") or xml_data.get(
"Title", f"wechat_file_{msg_id}"
)
logger.info(
f"WeChat file message: name={file_name}, media_id={media_id!r}, keys={list(xml_data.keys())}"
)
if media_id:
local, ann = await self._download_wechat_media(
media_id,
f"wechat_file_{msg_id}_{file_name}",
)
logger.info(f"WeChat file download result: local={local}, ann={ann}")
if local:
media_paths.append(local)
if ann:
annotations.append(ann)
if not media_paths:
annotations.append(f"[file: {file_name}]")
elif msg_type == "link":
title = xml_data.get("Title", "")
description = xml_data.get("Description", "")
url = xml_data.get("Url", "")
text = f"[链接] {title}\n{description}\n{url}"
elif msg_type == "event":
event_type = xml_data.get("Event", "")
if event_type == "subscribe":
text = "[用户关注]"
elif event_type == "unsubscribe":
logger.info(f"User {from_user} unsubscribed")
return # Don't process
elif event_type == "CLICK":
event_key = xml_data.get("EventKey", "")
text = f"[菜单点击] {event_key}"
elif event_type in ("LOCATION", "VIEW"):
# Periodic location reports and menu-link clicks — ignore
return
else:
logger.debug(f"Ignoring WeChat event: {event_type}")
return
else:
text = f"[{msg_type} message]"
if not text and not media_paths and not annotations:
return
# Parse timestamp
try:
timestamp = (
datetime.fromtimestamp(int(create_time))
if create_time
else datetime.now()
)
except (ValueError, TypeError, OSError):
timestamp = datetime.now()
await self._enqueue_raw(
RawIncoming(
sender_id=from_user,
chat_id=chat_id,
text=text,
media_files=media_paths,
content_annotations=annotations,
timestamp=timestamp,
message_id=msg_id,
is_group=is_group,
was_mentioned=was_mentioned,
metadata={
"chat_id": chat_id,
"to_user": to_user,
"backend": self._backend,
},
)
)
# ── Send (template method overrides) ──────────────────────────
def _format_chunk(self, text: str) -> str:
"""WeCom uses markdown formatter; MP uses plain text."""
if self._backend == "wecom":
return self._formatter.format(text) # markdown profile
return _strip_markdown(text)
async def _send_chunk(
self,
chat_id,
formatted_text,
raw_text,
reply_to,
metadata,
):
token = await self._ensure_token()
if self._backend == "wecom":
# Group chat: use appchat/send endpoint
if chat_id.startswith("wr"):
try:
await self._wecom_send_group_markdown(token, chat_id, raw_text)
return
except Exception:
pass
await self._wecom_send_group_text(token, chat_id, raw_text)
else:
# DM: Try markdown first, fall back to plain text
try:
await self._wecom_send_markdown(token, chat_id, raw_text)
return
except Exception:
pass
await self._wecom_send_text(token, chat_id, raw_text)
else:
await self._mp_send_text(token, chat_id, raw_text)
# ── WeCom send ────────────────────────────────────────────────
async def _wecom_send_text(
self,
token: str,
user_id: str,
text: str,
) -> None:
"""Send a text message via WeCom API."""
url = f"https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token={token}"
body = {
"touser": user_id,
"msgtype": "text",
"agentid": int(self.config.agent_id),
"text": {"content": _strip_markdown(text)},
}
await self._post_api(url, body)
async def _wecom_send_markdown(
self,
token: str,
user_id: str,
text: str,
) -> None:
"""Send a markdown message via WeCom API.
Note: WeCom markdown only supports a subset of Markdown
(no code blocks, no images). Falls back to text if the
message is too complex.
"""
url = f"https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token={token}"
body = {
"touser": user_id,
"msgtype": "markdown",
"agentid": int(self.config.agent_id),
"markdown": {"content": text},
}
await self._post_api(url, body)
# ── WeCom group send ────────────────────────────────────────────
async def _wecom_send_group_text(
self,
token: str,
chatid: str,
text: str,
) -> None:
"""Send a text message to a WeCom group chat."""
url = f"https://qyapi.weixin.qq.com/cgi-bin/appchat/send?access_token={token}"
body = {
"chatid": chatid,
"msgtype": "text",
"text": {"content": _strip_markdown(text)},
}
await self._post_api(url, body)
async def _wecom_send_group_markdown(
self,
token: str,
chatid: str,
text: str,
) -> None:
"""Send a markdown message to a WeCom group chat."""
url = f"https://qyapi.weixin.qq.com/cgi-bin/appchat/send?access_token={token}"
body = {
"chatid": chatid,
"msgtype": "markdown",
"markdown": {"content": text},
}
await self._post_api(url, body)
# ── MP send ───────────────────────────────────────────────────
async def _mp_send_text(
self,
token: str,
openid: str,
text: str,
) -> None:
"""Send a text message via WeChat MP customer service API."""
url = (
f"https://api.weixin.qq.com/cgi-bin/message/custom/send"
f"?access_token={token}"
)
body = {
"touser": openid,
"msgtype": "text",
"text": {"content": _strip_markdown(text)},
}
await self._post_api(url, body)
# ── Media send ────────────────────────────────────────────────
async def _send_media_impl(
self,
recipient: str,
file_path: str,
caption: str = "",
metadata: dict | None = None,
) -> bool:
"""Send a media file via WeChat/WeCom."""
token = await self._ensure_token()
chat_id = self._resolve_media_chat_id(recipient, metadata)
# Upload media to get media_id
media_id = await self._upload_media(token, file_path)
if not media_id:
return False
path = Path(file_path)
ext = path.suffix.lower()
is_image = ext in {".jpg", ".jpeg", ".png", ".gif", ".bmp"}
if self._backend == "wecom":
msg_type = "image" if is_image else "file"
# Group chat: use appchat/send endpoint
if chat_id.startswith("wr"):
url = (
f"https://qyapi.weixin.qq.com/cgi-bin/appchat/send"
f"?access_token={token}"
)
body = {
"chatid": chat_id,
"msgtype": msg_type,
msg_type: {"media_id": media_id},
}
else:
url = (
f"https://qyapi.weixin.qq.com/cgi-bin/message/send"
f"?access_token={token}"
)
body = {
"touser": chat_id,
"msgtype": msg_type,
"agentid": int(self.config.agent_id),
msg_type: {"media_id": media_id},
}
else:
url = (
f"https://api.weixin.qq.com/cgi-bin/message/custom/send"
f"?access_token={token}"
)
msg_type = "image" if is_image else "file" # MP only supports image
if not is_image:
# MP doesn't support file via customer service API;
# send caption as text instead
if caption:
await self._mp_send_text(
token, chat_id, f"[文件] {path.name}\n{caption}"
)
return True
body = {
"touser": chat_id,
"msgtype": "image",
"image": {"media_id": media_id},
}
await self._post_api(url, body)
# Send caption separately if provided
if caption:
if self._backend == "wecom":
if chat_id.startswith("wr"):
await self._wecom_send_group_text(token, chat_id, caption)
else:
await self._wecom_send_text(token, chat_id, caption)
else:
await self._mp_send_text(token, chat_id, caption)
return True
async def _upload_media(
self,
token: str,
file_path: str,
) -> str | None:
"""Upload a media file and return the media_id."""
path = Path(file_path)
ext = path.suffix.lower()
is_image = ext in {".jpg", ".jpeg", ".png", ".gif", ".bmp"}
media_type = "image" if is_image else "file"
if self._backend == "wecom":
url = (
f"https://qyapi.weixin.qq.com/cgi-bin/media/upload"
f"?access_token={token}&type={media_type}"
)
else:
url = (
f"https://api.weixin.qq.com/cgi-bin/media/upload"
f"?access_token={token}&type={media_type}"
)
try:
with open(file_path, "rb") as f:
resp = await self._http_client.post(
url,
files={"media": (path.name, f)},
)
data = resp.json()
if data.get("errcode", 0) != 0 and "media_id" not in data:
logger.error(f"WeChat media upload failed: {data.get('errmsg')}")
return None
return data.get("media_id")
except Exception as e:
logger.error(f"WeChat media upload error: {e}")
return None
# ── Media download helper ────────────────────────────────────
async def _download_wechat_media(
self,
media_id: str,
filename: str,
) -> tuple[str | None, str | None]:
"""Download media by media_id via WeChat/WeCom media API."""
token = await self._ensure_token()
if self._backend == "wecom":
url = f"https://qyapi.weixin.qq.com/cgi-bin/media/get?access_token={token}&media_id={media_id}"
else:
url = f"https://api.weixin.qq.com/cgi-bin/media/get?access_token={token}&media_id={media_id}"
return await self._download_attachment(url, filename)
# ── Shared API helper ─────────────────────────────────────────
async def _post_api(self, url: str, body: dict) -> dict:
"""POST to WeChat/WeCom API, check errcode, return response."""
try:
resp = await self._http_client.post(url, json=body)
data = resp.json()
except Exception as e:
raise RuntimeError(f"WeChat API error: {e}") from e
errcode = data.get("errcode", 0)
if errcode != 0:
errmsg = data.get("errmsg", "unknown")
# Token expired — refresh and retry once
if errcode in (40014, 42001):
logger.warning("WeChat token expired, refreshing...")
await self._refresh_token()
token = self._access_token
# Replace token in URL
if "access_token=" in url:
url = re.sub(
r"access_token=[^&]+",
f"access_token={token}",
url,
)
resp = await self._http_client.post(url, json=body)
data = resp.json()
if data.get("errcode", 0) != 0:
raise RuntimeError(
f"WeChat API error after retry: {data.get('errmsg')}"
)
return data
else:
raise RuntimeError(f"WeChat API error ({errcode}): {errmsg}")
return data
# ── Typing indicator (WeCom only) ────────────────────────────
async def _send_typing_action(self, chat_id: str) -> None:
"""Send typing indicator via WeChat.
WeChat has no native typing API. For the WeCom backend we
approximate the experience by posting a short-lived "…" message
that is recalled once the real reply is sent (handled by
``stop_typing``). WeChat MP has no recall API so we skip it.
"""
if self._backend != "wecom" or not self._http_client:
return
try:
token = await self._ensure_token()
if chat_id.startswith("wr"):
url = (
f"https://qyapi.weixin.qq.com/cgi-bin/appchat/send"
f"?access_token={token}"
)
body = {
"chatid": chat_id,
"msgtype": "text",
"text": {"content": "\u2026"},
}
else:
url = (
f"https://qyapi.weixin.qq.com/cgi-bin/message/send"
f"?access_token={token}"
)
body = {
"touser": chat_id,
"msgtype": "text",
"agentid": int(self.config.agent_id),
"text": {"content": "\u2026"},
}
data = await self._post_api(url, body)
msgid = data.get("msgid")
if msgid:
self._typing_message_ids.setdefault(chat_id, []).append(msgid)
except Exception:
pass
async def stop_typing(self, chat_id: str) -> None:
"""Cancel typing loop and recall all status messages."""
msgids = self._typing_message_ids.pop(chat_id, [])
if msgids and self._http_client and self._backend == "wecom":
try:
token = await self._ensure_token()
url = (
f"https://qyapi.weixin.qq.com/cgi-bin/message/recall"
f"?access_token={token}"
)
for msgid in msgids:
try:
await self._http_client.post(url, json={"msgid": msgid})
except Exception:
pass
except Exception:
pass
await super().stop_typing(chat_id)

Some files were not shown because too many files have changed in this diff Show More