anydoc converts the PDF text layer only and emits no image placeholders
or page markers, so a mostly-scanned PDF extracts 'successfully' into
section headers with empty bodies — silent data loss the model cannot
detect. Count per-page text via poppler pdftotext and prepend an
EXTRACTION COVERAGE WARNING naming the empty pages and the recovery
path (pdftoppm + vision_analyze, or the ocr-and-documents skill).
Found on a 311-page HOA resale package where 198 scanned pages
(CC&Rs, Bylaws, Articles, insurance certs) vanished without a trace.
Closes the last two emission gaps from #77484:
- tools/terminal_tool.py: both exception paths (generic except and
TERMINAL_DEGRADED_MODE=fail) returned raw str(e) + traceback.format_exc()
to the model — only the logger copy was redacted. Exception text can
embed the failing command line and any secrets inline in it; both fields
now pass through redact_sensitive_text.
- acp_adapter/entry.py: _setup_logging cleared root handlers and installed
a plain logging.Formatter, bypassing redaction entirely on ACP stderr.
Now uses RedactingFormatter like every other logging surface.
The other three gaps from #77484 (process(list), *_KEY regex variants,
control-char splits) were fixed in #80964/#80965.
image_generate and video_generate forwarded model-supplied local paths to
provider plugins, which read them off the HOST filesystem regardless of
terminal backend — inconsistent with the confinement boundary vision/video
analysis enforce (GHSA-gpxw-6wxv-w3qq), and broken for sandbox-only files.
New dispatch-layer chokepoint (_confine_source_images): under a non-local
backend, path-like image_url / reference_image_urls resolve through
tools.image_source (media-cache host reads, bounded in-sandbox exec-read,
lazy env bring-up, credential guard, 50MB cap) and reach every provider as
data: URLs — which all backends already accept. URLs/data: pass through;
local backend is a no-op. xai_video_edit/extend already require public
HTTPS URLs, so no change needed there.
Under the Docker terminal backend, vision_analyze's first exec-read
sometimes returned empty / non-zero against a freshly started container,
producing 'could not read <path> inside the sandbox' on a file the agent
could cat seconds later. Cold pipe setup on the first exec against a
new container, not a permissions or mount problem.
Retry once after a short delay (150 ms covers Docker exec warm-up
without making a real failure feel sluggish). When every attempt still
fails, fold the container's first stderr line into the raised error so
the user can tell 'no such file' from 'permission denied' instead of
staring at one opaque message.
Tests cover the retry-then-succeed path, the diagnostic-on-exhausted
path, and confirm the existing single-attempt raise is preserved.
Follow-up on the salvaged commit: replace the hand-rolled file_ops python3
exec-read with tools.image_source.resolve_image_source(permitted=('video',)),
so video_analyze gets the same pipeline as vision_analyze — media-cache host
reads, bounded head -c sandbox exec (no python3 dependency in the sandbox
image, no unbounded base64 stream), lazy env bring-up (#62825), the
credential-read guard, and the 50MB ingest cap.
The _bash_exec_payload delegation rejected short-option bundles with
letters outside bash's alphabet, so 'zsh -yc', 'dash -Vc' and 'ksh -Gc'
scripts stopped being scanned — a fail-open regression for shells the
guard's _SHELL_EXECUTABLES explicitly covers. Try the bash grammar
first (catches operand-hidden -c), then fall back to the permissive
positional scan; a block-guard fails closed.
bisect sat in _KNOWN_GIT_BUILTINS and was allowed in the running source
root, yet it repeatedly checks out commits — the exact module-version
skew this guard exists to prevent. Move it to _WORKTREE_MUTATIONS.
_KNOWN_GIT_BUILTINS omitted reset/stash/clean/restore (whose dangerous
forms _mutates_worktree already classifies first) and common read-only
porcelain (reflog, ls-files, cat-file, shortlog, show-ref, ls-tree,
ls-remote, merge-base). Every safe use inside the source repo — 'git
stash list', 'git reset --soft', 'git clean -n', 'git restore --staged'
— fell through to _read_git_alias and spawned a 'git config --get
alias.X' subprocess per terminal command (~10ms, 1s worst case on a
locked config). Complete the builtin set; the mutation classification
is unchanged and runs first.
The guard's _shell_script_arg treated any leading option containing 'c'
as -c and looked no further, so 'bash -o pipefail -c "git checkout
main"' returned None and the script was never scanned (fail-open).
approval.py's _bash_exec_payload already parses bash's real option
grammar (-O/-o consume operands, short-option bundles, --init-file);
delegate to it instead of keeping a second, weaker parser.
`worktree` sat in _KNOWN_GIT_BUILTINS, so the guard returned safe for the
whole family. That allowed `git worktree remove [--force] <root>` and
`git worktree move <root> <dest>` against the very checkout this process
runs from, which the guard already treats as a source root when its .git
is a linked-worktree file.
Both name their target as an argument rather than acting on the cwd, so
they also slipped past the "is the cwd inside root" gate when run from
outside. Resolve the target against the command's cwd and block it when
it lands on the running root, from any directory. `worktree add`, list,
prune, lock, unlock, and operations on other worktrees stay allowed.
Wire the self-repo guard in next to the gateway lifecycle hard-block,
before the force check — force=True cannot make the command safe, only
delay the crash. Local backend only: sandboxed backends cannot reach the
host checkout. The block message explains the version-skew mechanism and
redirects to git worktree add / a temp clone, or running the command
outside hermes with a restart after.
When hermes runs from a source/editable install, a git checkout/reset/pull
in its own repo swaps code on disk under the live interpreter. Modules
imported before the switch stay old while later lazy imports load new code,
producing delayed signature TypeErrors and tracebacks that don't match the
source, typically losing the in-flight turn.
New tools/self_repo_guard.py detects working-tree/ref mutations (checkout,
switch, reset, rebase, merge, pull, restore, stash, clean, cherry-pick,
revert) whose target repo is the source root the process runs from, via
cwd, git -C, cd chains, and subshell segments. Read-only git, commits,
fetch, and git worktree add stay allowed; packaged installs (no .git) are
inert.
dispatch() called logger.exception with the exception interpolated into the
message. exc_info renders the same exception again in the traceback, so a
failing tool wrote its error body to the log twice. Every tool exception
passes through this one handler, so a large HTTP error body from any tool
landed here at full size.
Bound the message copy. The traceback still renders the exception once,
which is what an operator needs to place the failure.
Same double-write @arimu1 fixed in the vision, image, and TTS handlers in
#75938.
The debug line that fires when an error body is truncated interpolated the
whole untruncated body, so capping the model-facing copy still wrote the
original to the log. A large HTTP error body — a Cloudflare challenge page
or a proxy 502 — reached the log at full size on every failed call.
Log a bounded prefix instead. It stays longer than the model-facing cap so
an operator still has something to diagnose with, but it no longer grows
with the size of the response body.
Reported for the logging handlers in #75938 by @arimu1; the same pattern
was present here.
tool_error() caps its message at _MAX_TOOL_ERROR_CHARS (2048), logging
the full body at DEBUG before trimming the context-bound copy.
Handlers that serialize exceptions directly -- json.dumps({"error":
str(exc), ...}) -- bypass that helper, so _normalize_handler_result
also runs every string result through _bound_json_error_result: if it
parses as a JSON object with an oversized string error field, only
that field is trimmed and the payload re-serialized. Non-error
results, non-JSON strings, and multimodal envelopes pass untouched.
Open the selected microphone at its reported default rate and convert each capture block to the 16 kHz frame expected by wake-word engines. Add a regression covering a 48 kHz WASAPI device.
Co-authored-by: clyu168 <clyu168@126.com>
Two claim-failure diagnostic paths (cronjob_tools.py:629,921) still used
the old inline 'not enabled or state==paused' check. After get_job()
normalizes via effective_job_state, a half-paused record has
state='scheduled' and enabled=True, so the inline check returned False —
mislabeling the job as 'already being fired' instead of 'paused/disabled'.
Also hoists effective_job_state/is_job_runnable to the top-level import in
cronjob_tools.py (was function-local) and updates console_engine.py's
_format_job to use effective_job_state instead of the old inline
state-or-enabled derivation — a fourth display path the original PR missed.
Follow-up to PR #81287.
pause_job already sets enabled=false atomically with state/paused_at, but
get_due_jobs only checked enabled — so a contradictory record
(enabled=true + paused_at/state=paused) still fired. That was the 07-30
outage failure mode: list looked frozen, fleet kept merging.
- is_job_runnable / effective_job_state: pause markers gate fire; display
derives from the scheduler-honoured enabled flag so half-paused never
renders as [paused]
- get_due_jobs self-heals enabled=false + logs error on contradiction
- claim_job_for_fire uses is_job_runnable (paused_at counts too)
- list/format paths use effective_job_state
- behavioural tests: pause blocks due fire; half-pause self-disables
An auth failure on the very first connect returned out of the run loop
instead of parking. That ended the run task, and the task is the only
listener on _reconnect_event — so the server stayed dead for the life of
the process. `hermes mcp login`, a /mcp refresh, and the 300s self-probe
all had nothing left to wake, and the only cure was a full restart.
_classify_mcp_failure already calls 401/403 "permanent" and documents
that run() parks those immediately; the early return above it meant auth
was the one permanent failure that never got there. Park it with the
others and keep the tailored log line, now pointing at `hermes mcp
login <server>`.
When Hermes runs as a systemd gateway with MemoryHigh/MemoryMax limits,
local background terminal commands (terminal(background=true)) inherit the
gateway's cgroup. A memory-heavy executor (Codex, tests, Node) can push
the whole cgroup past MemoryMax and trigger systemd-oomd to kill the
ENTIRE gateway — taking down the messaging control plane and silently
losing the active turn.
Root cause: tools/process_registry.py::spawn_local() uses
start_new_session=True (creates a process session/group, NOT a resource
cgroup). The spawned process tree stays in the gateway's systemd cgroup.
Fix: when running under a service manager (detected via the existing
is_gateway_supervisor_process() helper), wrap the pipe-mode spawn command
in 'systemd-run --user --scope --unit=hermes-worker-<id>' so the worker
gets its own transient cgroup. An OOM in the worker then kills only the
worker, not the gateway.
The systemd-run availability is probed once (a no-op /bin/true in a
transient scope) and cached, because the binary can exist on PATH while
the user D-Bus session is unavailable (system services, containers). If
unavailable, fall back to the current start_new_session=True behavior
with a debug log.
Scope: this covers the common background pipe-mode path. PTY mode
(PtyProcess.spawn) is left as future work — it uses a different spawn
mechanism and is used for interactive CLI tools where cgroup isolation
has additional considerations.
vision_analyze reads container-only images by exec-reading them inside the
sandbox, but unlike terminal_tool it never triggered environment creation. Under
a non-local backend (ssh, docker, ...), a session whose first action was
vision_analyze on a remote path failed with 'no active sandbox session' until an
unrelated terminal command happened to establish the connection.
Add terminal_tool.ensure_task_env(task_id), a public lazy get-or-create that
reuses the terminal tool's own creation machinery, and call it from
image_source._resolve_container_fallback before the in-sandbox read. Extract the
ssh/container config-dict builders so both paths derive settings identically
(no duplication). Best-effort and fail-closed: a failed bring-up leaves the
existing 'no active sandbox' error intact, never a host read.
Fixes#62825
Connection-class infrastructure failures on remote terminal backends (SSH
host unreachable/timed out, Docker daemon down or missing, remote file
sync failing on a dead link) previously surfaced to the model as raised
RuntimeError tracebacks. The model got a stack blob with no guidance and
the failure was indistinguishable from a tool bug.
Now:
- New EnvironmentConnectionError(RuntimeError) in tools/environments/base.py
carrying a reason + retry_hint. Subclassing RuntimeError keeps every
existing catcher working.
- ssh.py classifies connect-refused, connect-timeout, scp, remote mkdir,
bulk upload/download, and remote rm failures as connection errors.
- docker.py classifies all four _ensure_docker_available() failure paths
(missing exe, non-executable exe, daemon timeout, `docker version`
failure).
- terminal_tool catches EnvironmentConnectionError and returns a
structured tool result the model can act on:
{"status": "degraded", "reason": ..., "retry_hint": ..., "exit_code": -1}
The failed backend is evicted from the environment cache so a later
call retries from scratch — recovery is automatic once the backend is
reachable again.
- Config gate terminal.degraded_mode: warn|fail (default warn) in
config.yaml, bridged as TERMINAL_DEGRADED_MODE across all four bridge
sites (cli.py env_mappings, gateway/run.py _terminal_env_map,
TERMINAL_CONFIG_ENV_MAP, DEFAULT_CONFIG). "fail" preserves the
historical error+traceback tool result.
- Command failures (nonzero exit, command-not-found) are NOT touched —
only infrastructure failures classify as degraded.
Tests: tests/tools/test_terminal_degraded_mode.py (15 tests) covering
exception classification for ssh+docker, structured degraded results,
no-caching of degraded envs, recovery after the backend returns,
nonzero-exit results unaffected, fail-mode preservation, invalid-mode
fallback to warn, and the four-site config bridge invariant.
Inspired by: Claude Cowork degraded-backend behavior (idea-level,
docs-only evidence).
Per-task `output_schema` (JSON Schema object) on task items plus the
top-level single-goal form — a one-time static addition to the tool
schema (never varies per call).
- Child side: the schema is appended to the child's context as an
explicit OUTPUT CONTRACT block before spawn.
- Completion side: the parent validates the child's final answer with
jsonschema; on failure it sends exactly ONE bounded retry turn
carrying the validation errors verbatim (no schema re-paste).
- Result entries gain schema_valid (+ schema_retries, and schema_errors
on final failure) ONLY when a schema was requested; schema-less calls
keep a byte-identical result shape.
- Malformed schemas are rejected loudly at dispatch (coerce_output_schema
meta-validates via jsonschema's validator_for/check_schema).
- New helpers in tools/delegation_output_schema.py: coerce, contract
block, fence/prose-tolerant extraction+validation, retry message.
Pattern from: github/copilot-cli ctx.agent(prompt,{schema}) — PATTERN
ONLY, zero code/prompt text copied (proprietary); proven consumer:
delegate-task-output-patterns skill.
Tests: tests/tools/test_delegate_output_schema.py (24 tests — valid
first try, invalid->retry->valid, invalid twice -> schema_valid false +
errors surfaced, retry-exception degrade, no-schema legacy shape pin,
dispatch rejection, contract plumbing). Delegation suite: 221/221 green.
Add an optional `region: [x1, y1, x2, y2]` parameter to vision_analyze
(pixel coordinates in the ORIGINAL image space). The crop is applied
with Pillow BEFORE the downscale/embed-cap pipeline, so the cropped
region gets the full resolution budget — a zoom for reading small text
or UI details after a full shot.
- New `_crop_image_region` helper: clamps out-of-bounds coordinates to
the image, rejects zero-area/inverted/malformed regions with an error
naming the actual image dimensions so the model can retry sensibly.
- Wired into both the native fast path (`_vision_analyze_native`) and
the legacy aux-LLM path (`vision_analyze_tool`).
- Schema gains one static optional param (byte-stable thereafter); the
description documents the intended flow: full shot first, then zoom.
- No region supplied = behavior unchanged (regression-guarded).
Tests: tests/tools/test_vision_region.py (11 tests — crop applied,
clamping, zero-area rejection with dims, malformed input, pre-downscale
full-budget zoom, schema shape, handler pass-through, no-region
unchanged). Widened one narrow fake_native stub in test_vision_tools.py
to be kwargs-tolerant.
Ported from: QwenLM/qwen-code zoom-image.ts (Apache-2.0)
write_file/patch targeting AGENTS.md, CLAUDE.md, SOUL.md, .cursorrules, or a
project-local .hermes config dir now ALWAYS prompt the human for approval —
even under --yolo/auto-approve — and fail closed when no human channel
exists. These files steer future agent behavior, so an injected write to
them is a prompt-injection persistence vector.
Design:
- New _check_protected_instruction_write() in tools/file_tools.py, a
sibling of _check_sensitive_path that returns approval-required rather
than a hard error. It realpaths before matching (symlink lesson from
#41351), matches basenames case-insensitively in ANY directory, rejects
'./x/../AGENTS.md' traversal via normpath, and gates files whose
immediate parent dir is `.hermes` (project-local config) while exempting
the authoritative ~/.hermes home (governed by its own guards).
- Approval is ONE-OPERATION only: no session/permanent persistence, no
yolo bypass — intentionally does not route through _run_approval_gate.
Gateway sessions get the button round-trip with allow_permanent and
allow_session both False; CLI uses the per-thread approval callback;
no channel at all = BLOCKED (fail closed).
- Multi-file V4A patches: ONE protected file gates the ENTIRE patch (a
single prompt lists all protected targets; deny applies nothing).
- Config: security.protected_instruction_files (default true) and
security.protected_instruction_extra_patterns (fnmatch on basename).
Config read failure keeps the gate ON.
Tests: 22 new cases in tests/tools/test_file_write_safety.py covering the
adversarial checklist — deny/approve/yolo-bypass attempt, symlink at a
protected target, case variants, relative traversal, arbitrary-directory
basenames, project-local .hermes, checkout-nested-under-~/.hermes
non-gating, patch replace + V4A multi-file atomicity, gateway round-trip,
fail-closed with no human, config off/extra patterns.
Ported from: RooCodeInc/Roo-Code RooProtectedController (Apache-2.0);
companion: #58631 (terminal vector), symlink lesson from #41351.
Adds a per-server `trust: full|untrusted` config key
(mcp_servers.<name>.trust). On an untrusted server, every write-capable
tool call — any tool whose discovery-time annotations do not carry
readOnlyHint=True — routes through the existing approval surface
(tools.approval.request_elicitation_consent, same lazy-import +
surface-routing pattern the MCP elicitation handler uses) before the RPC
fires. Denied/cancelled/errored approvals fail closed: the RPC never
runs, including the lazy first-use server spawn.
Design points:
- Classification happens at CALL TIME from metadata captured at
DISCOVERY (_record_tool_trust_metadata in _register_server_tools and
the lazy cache-registration path). No toolset/schema mutation, so the
toolset stays byte-stable and prompt caching is preserved.
- readOnlyHint is a server-supplied HINT: on an untrusted server a lying
server can at most skip approval for tools it claims read-only — it
can never widen access. Trust tiering itself is operator config.
- Missing/malformed annotations => write-capable (fail closed).
- Unrecognized trust values => untrusted (fail closed); missing key =>
full (backward compatible, documented in mcp-config-reference).
- The schema cache now persists readOnlyHint so lazy-registered servers
gate identically on next startup without spawning.
Tests: tests/tools/test_mcp_trust_gating.py (11 tests, TDD red->green):
approval invoked + accept proceeds, deny/cancel blocks RPC, readOnlyHint
=true skips gate, trusted/unconfigured servers skip gate, explicit
readOnlyHint=false gated, approval exception fails closed, trust
normalization, discovery-time capture (SDK objects and cached dicts).
Ported from: cloudflare-os classifyTool() (Apache-2.0), corroborated by
Claude Cowork (idea-level).
MCP servers generated from Rust/TypeScript union types commonly emit
closed value sets as const unions:
{"anyOf": [{"const": "red"}, {"const": "green"}, {"const": "blue"}]}
Strict tool-calling backends reject or mishandle these; the equivalent
property-level enum form is universally supported. Add
collapse_const_unions() to tools/schema_sanitizer.py and wire it into
the _normalize_mcp_input_schema discovery pipeline after the nullable
strip.
Rules:
- Collapse only when EVERY non-null branch is a pure const of the same
primitive type (bool never merges with integer).
- Mixed unions, non-uniform const types, and mismatched declared types
pass through untouched.
- A single {"type": "null"} branch is tolerated: consts -> enum,
null -> nullable: true hint (matches strip_nullable_unions, which
leaves null+multi-const unions alone by its one-non-null-branch rule).
- Outer title/description/default/examples carried onto the replacement.
- Deterministic, branch-order-preserving, non-mutating — applied at
discovery only, so schemas stay byte-stable per conversation.
Ported from: block/goose tool_schema_normalize.rs (Apache-2.0)
Two small config-gated features:
1. Kanban orphaned-card reconciliation (kanban.reconcile_orphans, default
true, config.yaml): a running card with broken claim bookkeeping
(claim_lock or claim_expires NULL — crash mid-claim, manual SQL, DB
restore) is invisible to all existing recovery paths
(release_stale_claims requires claim_expires NOT NULL,
detect_crashed_workers requires host-local lock + pid,
detect_stale_running is config-disabled by default) and shows Running
forever. New reconcile_orphaned_running() pass in kanban_db.py runs
each dispatch_once tick: requeues orphans to ready with an explanatory
comment, closes any leaked run, emits a 'reconciled' event, and defers
when the recorded PID is still alive on this host (never requeue
beside a live worker). Surfaced via DispatchResult.reconciled_orphans.
2. Per-server MCP identity header (mcp_servers.<name>.identity_header,
config.yaml): optional {name, value_from: static|profile, value}
mapping; the header is attached to that server's HTTP/SSE transport
requests. 'static' sends the config value; 'profile' resolves the
active Hermes profile name once at connect time (no per-call
mutation). Explicit per-server headers of the same name (any casing)
win. Invalid blocks warn-and-ignore; stdio servers warn-and-ignore.
Tests: tests/gateway/test_kanban_reconcile_orphans.py (9),
tests/tools/test_mcp_identity_header.py (13), all written first (RED)
then implemented (GREEN). No new HERMES_* env vars.
Inspired by: openai/symphony tracker reconciliation (Apache-2.0) +
Poke per-user MCP identity (idea-level).
Each serialized result entry now carries cost_usd (rounded to 6 dp)
and cost_status (the child's session_cost_status — 'estimated',
'reported', 'included', or 'unknown') alongside tokens/api_calls/
duration, so the parent model can see what each delegation cost.
The internal _child_cost_usd field is still stripped before
serialization and the parent session cost rollup is untouched.
Tool schema is unchanged (byte-stable).
Inspired by: Perplexity Agent API result shape (idea-level)
Reject malformed tasks=[...] batches before any child agent is spawned:
- exact-duplicate goals (case/whitespace-normalized), error names both
task indices
- placeholder goals: bare 'TODO', bare 'task N', unexpanded <...> or
{...} template markers, or goals shorter than 10 chars after strip
- 1-task batches, with an error pointing the model at the single
`goal` form instead
All checks are batch-only — the single-goal form is exempt by design
(short goals like goal="test" are valid there). Error strings are
actionable: each tells the model exactly how to fix the call.
Tool schema is unchanged (byte-stable); validation is runtime-only in
the existing batch-validation region.
Existing tests using terse batch goals ("A"/"B"/"C") updated to
realistic distinct goals per the new contract.
Inspired by: MoonshotAI/kimi-code agent-swarm.md validation rules (MIT)
Add monitor-mode cron jobs: a cheap monitor source (monitor_script or
monitor_url) runs on every tick BEFORE any agent machinery is built.
Its output is hashed as exact bytes and compared to the hash stored
from the last agent-triggering tick:
- unchanged -> agent run suppressed entirely (no LLM, no delivery);
the tick is recorded as a silent no_change run visible in the
executions ledger doc
- changed -> a MONITOR CHANGE DETECTED block (capped unified diff of
previous vs current output + the new output) is injected into the
prompt via the existing extra_prompt seam, then a normal agent run
- first run -> always runs the agent with a baseline block
- source failure -> delivered as an ERROR alert, never treated as a
change; the stored hash is untouched so recovery to prior output
still suppresses
Implementation:
- cron/monitor.py (new): hash/diff/URL-fetch/state persistence.
monitor_script reuses _run_job_script (same ~/.hermes/scripts/
containment + interpreter rules); monitor_url is a bounded GET
(30s, 256KB, http/https only). Output is exact bytes by design —
scripts should emit stable output (documented).
- cron/jobs.py: additive job fields monitor_script / monitor_url /
monitor_state {last_output_hash, last_changed_at}. JSON job records
need no migration. create-time validation: sources are mutually
exclusive and incompatible with no_agent=True.
- cron/scheduler.py: one tight monitor gate in run_job between the
no_agent short-circuit and the LLM path (outside sibling-lane
regions). State persists in jobs.json + a per-job snapshot file, so
suppression survives scheduler restarts.
- tools/cronjob_tools.py: additive optional monitor_script/monitor_url
params on the cronjob tool (create + update, empty string clears),
path containment validated at the API boundary, surfaced in
_format_job.
- hermes_cli: --monitor-script/--monitor-url on `hermes cron create`
and `hermes cron edit`; `hermes cron list` shows the monitor source
and last-changed time.
Tests (tests/cron/test_monitor_kind.py, TDD): unchanged suppresses,
changed injects diff, first run always runs, hash persists across
module reload (restart), script failure is error-not-change with hash
untouched, create/update validation, tool wiring + path-escape reject.
Inspired by: ChatGPT Work monitor tasks (idea-level, docs-only);
enabler: #80774
Review pass on the idle-unload feature found two material concurrency
bugs; both fixed here with a regression guard:
1. Unload-vs-use null deref (HIGH): _transcribe_local re-read the
module global _local_model at the transcribe call site. An idle
unload firing between the model load and transcribe() evaluated
None.transcribe → AttributeError → user-visible 'Local
transcription failed'. The window was real: the idle timer was only
touched AFTER a successful transcription, so a voice note arriving
exactly as the timeout expired raced the watcher directly.
Fix: bind a strong local reference under the model lock and use it
for the whole transcription (the watcher can null the global at any
time; this in-flight call keeps its instance — the generator holds
self, so no use-after-free). Also touch the idle timer at the START
of transcription so a long in-flight transcribe can't be counted as
idle time. The CUDA-fallback retry path gets the same treatment
(locked global write, local ref use).
2. Watcher replacement race + response-path join (MEDIUM/HIGH): the
old design stopped and re-started the watcher after EVERY
transcription with an unlocked set/join(5)/clear/start sequence on
shared globals. Two concurrent voice messages could interleave to
leave TWO live watchers (one with a stale, shorter timeout — a
raised unload_after_idle_seconds could still unload on the old
value), and the join(timeout=5) sat on the user-visible response
path (a watcher blocked on _local_model_lock during a concurrent
multi-second model load stalls the reply up to 5s).
Fix: single long-lived watcher under a management lock — started
only when none is alive (per-transcription cost: one lock + one
is_alive check), re-reads the configured timeout from config every
cycle (config edits now apply within one 30s interval, without
waiting for the next voice message — previously undocumented), and
stands down without unloading when the timeout is set to 0
mid-idle.
Tests: 17 now — idempotent start (same thread, no churn), config
re-read + stand-down-when-disabled, and the race guard
(unload firing mid-transcription must not fail the in-flight call).
The race guard is mutation-verified: reverting the fix (re-reading
the global at the call site) makes it fail with the exact NoneType
error; the fixed code passes.
The local faster-whisper model singleton (_local_model) is loaded once
and never released — the 'base' model holds ~370 MB of RAM/VRAM for
the entire lifetime of the process, even when no voice messages arrive
for hours or days. On long-running gateway processes (especially with
local LLMs competing for the same GPU) this is wasteful.
Add a config-driven idle unload: after stt.local.unload_after_idle_seconds
(default 0 = never) of no transcription activity, a lightweight daemon
thread sets _local_model = None so the Python GC can reclaim the
ctranslate2 objects. The next voice message reloads the model
transparently (the existing lazy-load path handles it).
The watcher:
- Checks every 30s whether idle time exceeds the configured threshold
- Acquires _local_model_lock before unloading (prevents races with
concurrent transcriptions that are mid-load)
- Exits immediately if the model is already None (unloaded by another
path, e.g. the CUDA fallback eviction)
- Is restarted by each transcription with the current config value,
so changing stt.local.unload_after_idle_seconds in config.yaml takes
effect on the next voice message without a process restart
Default is 0 (never unload) — zero behavior change for existing users.
Recommended value for gateway processes: 300 (5 minutes).
15 tests: config resolution (garbage/negative/None fallbacks), unload
safety (already-None, lock acquisition), touch timestamp, watcher
lifecycle (unload after timeout, no unload within timeout, exits when
model already None, stopped on new start). Existing STT test suite
unchanged.
Three-reviewer pass (reuse / quality / efficiency) on the trim diff;
four findings folded:
1. Short-clip input gate (efficiency, HIGH): the trim previously paid
the full ffmpeg encode before the <10%-saving discard check — every
dense conversational voice note burned 3 subprocess spawns + a
complete re-encode on the synchronous response path for nothing.
New _CLOUD_TRIM_MIN_INPUT_SECONDS=12 gate: below it, savings can't
matter (a >=10% saving is ~1s of audio, and several providers bill
a per-request minimum anyway — Groq bills 10s minimum), so the
whole pipeline is skipped using the duration we already probed.
Typical 5-10s voice notes now pay 1 ffprobe (~50ms), not 3 spawns +
encode (~0.3-1s; multi-second on small-VPS gateway hosts).
2. Shared encode profile (reuse, HIGH): the trim's ffmpeg command
duplicated _transcode_audio_for_stt's encode byte-for-byte (same
16kHz/mono/AAC-32k/faststart args, same subprocess.run kwargs).
Extracted _STT_M4A_ENCODE_ARGS + _run_ffmpeg_stt_encode(ffmpeg,
in, out, audio_filter=None); both call sites now share one owner,
so codec/bitrate/timeout changes can't drift between the paths.
3. is_truthy_value for the enable flag (quality, MEDIUM): raw
bool(cfg.get(...)) treated a YAML string "false" as enabled — the
exact bug class utils.is_truthy_value (already imported, already
used by is_stt_enabled and the xai/elevenlabs flags) exists for.
4. All-silence guard scales with keep_ms (quality, LOW): the fixed
0.3s floor equals the default keep window, so an output consisting
solely of one kept pause could pass as "speech"; now
max(0.3, 2*keep_seconds).
Also: _probe_audio_duration docstring documents it as the canonical
sync seconds-probe (gateway/run.py and the Telegram adapter carry
local variants of the same ffprobe invocation).
Tests: 24 now — YAML-string-false disables; short clips skip the
encode entirely (encoder mock asserted not-called); E2E fixtures
moved past the input gate. E2E re-verified: 13.2s note -> 6.2s
(-53%), 8s clip skipped with 1 probe.
Local faster-whisper gets Silero VAD (bf8004e3a) so silence never
reaches the model. Cloud providers got no such protection: the raw
file uploads untouched, so every second of silence in a voice note is
paid for twice — upload time and per-audio-minute billing — and cloud
Whisper hallucinates junk tokens on silent stretches exactly like
local Whisper did before the VAD hardening. A 13s voice note with two
long pauses is billed as 13s of audio to transcribe ~6s of speech.
Close the gap client-side: before uploading to a built-in cloud
provider (groq/openai/mistral/xai/elevenlabs/deepinfra), collapse long
pauses with ffmpeg's silenceremove filter, keeping
stt.cloud_trim_keep_ms (default 300) of every pause so word boundaries
and natural pacing survive. Uses ffmpeg, already a dependency of this
exact path via _transcode_audio_for_stt — no new dependency.
The trim is strictly best-effort — ALL of these upload the original
untouched, transcription never fails because of the trim:
- stt.cloud_trim_silence: false
- ffmpeg/ffprobe missing, trim failure, or timeout
- trimmed result ~empty (mostly-silence clip: the provider, not a
client-side dB heuristic, decides whether it contains speech)
- trim saves <10% (re-encoding for nothing)
Command-type and plugin providers are deliberately NOT trimmed: they
may wrap local CLIs that want the original bytes or run their own VAD.
E2E (real ffmpeg + faster-whisper): 13.2s voice note with 7s pause ->
6.2s upload (-53%); transcript of trimmed audio matches the original
on both utterances. Dense-speech and all-silence WAVs correctly fall
back to the original. 22 unit+E2E tests; STT/voice suite failures
identical to upstream/main baseline (all pre-existing).
_download_image() and _download_video() both used client.get() +
response.content, buffering the entire media body into memory before
checking the size cap. A server that omits Content-Length could send
an arbitrarily large payload, causing OOM.
Extract _stream_download_to_file() shared helper: streams via
client.stream() + aiter_bytes(), writes chunks to a temp file, enforces
the running byte count against the cap after each chunk, and atomically
replaces onto the destination on success. Cleans up the temp file on
failure. Uses utils.atomic_replace() for cross-device/symlink safety.
Malformed Content-Length values are now caught and ignored instead of
crashing with ValueError; the streaming cap is the authoritative guard.
Approach adapted from PR #10440 by @WuKongAI-CMU (closed as stale —
14923 commits behind, reverted 32 commits of vision_tools.py evolution
including SSRF-safe client, retry classification, and lazy imports).
Closes#10440
Teknium