Commit Graph

2766 Commits

Author SHA1 Message Date
Jash Lee 63954d508c fix(agent): recover from dropped tool calls (finish_reason=tool_calls, empty array)
Some providers (observed: claude-opus-4.8 / claude-sonnet-4.5 on GitHub
Copilot, ~2026-07) return finish_reason="tool_calls" while the parsed
tool_calls array is empty — the model signalled it wanted to act but the
payload shipped no call. The conversation loop took the no-tool-calls
else branch, treated the turn's narration as the final answer, and exited
with the task unstarted.

On unattended multi-step jobs this is silent failure: a scheduled PR
reviewer, for example, would narrate "Let me verify the PR..." and stop
at tool_turns=0 every run, never submitting a review, while the job still
reported success.

Fix: in the no-tool-calls branch, detect the provider contract violation
(finish_reason == "tool_calls" with zero tool_calls) and re-prompt the
model to emit the call instead of exiting. Bounded to 3 consecutive
stalls; the budget resets after any successful tool round so it guards
each stall independently rather than capping the whole run. The narration
may live in content or only in the reasoning field (empty content) — the
guard keys on the finish_reason/tool_calls mismatch, so both are covered.
A genuine finish_reason="stop" text turn is unaffected.

Verified live: a scheduled reviewer that died at tool_turns=0 every run
now recovers through 20+ stalls per run and submits real reviews.

Tests: tests/run_agent/test_dropped_tool_call_recovery.py covers the
re-prompt, the empty-content case, the clean-stop control, and the
bounded-loop guard.
2026-07-24 15:56:39 -07:00
webtecnica 6a9340d40a fix(agent): mark tool failures in the activity log (#69131)
The last_activity field showed 'tool completed: read_file (120.7s)'
identically whether the tool succeeded or failed, making post-mortem
analysis of silent-hang reports (#69131) unnecessarily hard.

Append ' (error)' to the activity description when the tool result is
classified as a failure, in both the concurrent and sequential
execution paths.

(Salvaged from PR #69467; its conversation_loop.py activity-touch hunk
is the same fix as PR #69577 and lands in the preceding commit.)
2026-07-24 15:56:09 -07:00
webtecnica 182c09b80b fix(agent): prevent agent hang after tool call completes
After a tool call completes, the conversation loop does  to
start the next API call. Between the last  (from tool
completion in tool_executor.py) and the next one (at the start of the
next API call), there is a gap. If anything during this gap takes time
— context compression, slow provider prefill, or other post-tool
processing — the combined inactivity window can exceed the gateway's
inactivity_timeout (default 120s). The gateway kills the session and the
user sees 'agent never returns a final response', even though the tool
call itself succeeded in 0.1-0.2s.

Fix: call  right before  so the gateway
sees a fresh timestamp immediately after tool results are posted,
regardless of how long the follow-up API call takes.

Fixes #69559
2026-07-24 15:56:09 -07:00
Dhruv Raajeev d10d3d7b42 fix(gateway,tools,agent): close leaked SQLite connections in delivery, delegation, and verification ledgers
Three durable ledgers used `with _connect() as conn:` where the sqlite3
connection context manager commits/rolls back but never closes, leaking the
db/-wal/-shm file descriptors on every call. On a long-running gateway this
exhausts RLIMIT_NOFILE and fails unrelated components with
`[Errno 24] Too many open files`. Same bug class as the cron execution ledger
(#69567 / PR #69594), which the connection helpers here are modeled on.

Fix: route every ledger operation through a `_transaction()` context manager
that guarantees `conn.close()` on exit. `_connect()` keeps its
schema-on-connect contract (several tests call it directly) and now self-closes
if schema init fails.

Adds per-module regression tests asserting every opened connection is closed,
including the no-op-update and exception-mid-transaction paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 15:55:08 -07:00
teknium1 dc6cb5c500 fix(anthropic): keep replay content schema-valid when every block is blank
Follow-up to the cherry-picked #68633 commits, closing the final open
review point (egilewski): _relocated_replay_cache_control was applied
only inside `if replayed:`. When anthropic_content_blocks contained
only a blank cache-marked text block, `replayed` came out empty, the
function fell through to the main path's placeholder, and the cache
marker was lost; signed thinking + a blank marked text block likewise
returned with no cacheable carrier for the relocated marker.

The replay branch now appends the non-whitespace "(empty)" placeholder
when no cacheable (text/tool_use) block survives the blank filter and a
blank text block was dropped (or a marker needs a carrier) — so replay
stays schema-valid on Bedrock/strict endpoints and the breakpoint
survives on the placeholder.

Also reconciles the block-level tests from #69517 with the new
drop-then-fallback contract (blank blocks are dropped at the block
level; the message-level result is still always non-blank).

Refs #69512

Co-authored-by: ygd58 <buraysandro9@gmail.com>
2026-07-24 15:54:38 -07:00
ygd58 29f9cfeb4a fix(anthropic): don't fall back to raw content when all blocks filtered
Follow-up per independent review of #68633 (GPT-5.6-sol-xhigh in Codex,
reviewer egilewski) on this PR.

Two real bugs in the blank-text-block filtering added by that fix:

1. `effective = blocks or content` fell back to the RAW, unfiltered
   `content` variable whenever every block was filtered out as blank --
   which happens precisely when the entire message content WAS the
   blank/whitespace payload the filter exists to remove (a sole blank
   text block, a sole cache-marked blank block, or standalone
   whitespace scalar content with no tool_calls). The fallback silently
   restored the exact invalid content the filtering just stripped,
   leaving the message provider-invalid.

   Fixed: `effective = blocks if blocks else [{"type": "text", "text":
   "(empty)"}]` -- never falls back to raw `content`. Also moved the
   cache_control application (both the relocated-from-a-dropped-block
   marker and the message-level marker) to run against `effective`
   instead of the pre-fallback `blocks`, so a cache marker on a block
   that was the ONLY content still lands on the (empty) placeholder
   rather than being silently lost when `blocks` was empty at the
   point it would otherwise have been applied.

2. The normal-path blank-text check used `(blk.get("text") or "").strip()`,
   which is not type-safe for a truthy NON-string, non-None text value
   (e.g. an int or dict from an invalid upstream payload) -- `or`
   doesn't substitute for a truthy value, so `(7 or "").strip()` still
   raises AttributeError. Now checks `isinstance(text, str)` first,
   matching the replay path's `_sanitize_replay_block()`, which the
   reviewer confirmed was already correctly type-safe.

Added regression tests for: sole blank list block, sole whitespace
scalar content, sole cache-marked blank block (marker relocation to
the placeholder), a truthy non-string (int) text value both mixed with
a surviving tool_use and as the sole content, and a dict-valued text
field. 7/7 new tests pass; 193/193 in the full
tests/agent/test_anthropic_adapter.py file; 23/23 in
tests/agent/test_prompt_caching.py (unaffected, confirmed).
2026-07-24 15:54:38 -07:00
ygd58 c55d780d48 fix(anthropic): filter blank text blocks in both normal and replay paths
Ports #63228 forward onto current main per teknium1's review.

Bedrock and strict Anthropic-compatible endpoints reject text blocks
where text is empty or whitespace-only with HTTP 400. The normal
list-content path extended blocks without filtering, and the
ordered-replay fast path (_sanitize_replay_block) returned blank text
blocks unfiltered.

Per review, fixes three gaps in the original port:

1. Type safety: the normal-path filter used blk.get('text', '').strip(),
   which crashes with AttributeError when text is explicitly None (not
   absent) -- .get()'s default only applies when the key is missing.
   _convert_content_part_to_anthropic() can preserve None from an
   invalid upstream input text block. Now uses
   (blk.get('text') or '').strip() on both paths.

2. Cache marker loss: prompt_caching.py's _apply_cache_marker() sets
   cache_control directly on content[-1] for list content. If that last
   part happens to be blank text, dropping it without relocating
   cache_control silently loses the breakpoint. Both the normal and
   replay paths now capture a dropped block's cache_control and reapply
   it to the new last surviving cacheable block via the existing
   _apply_assistant_cache_control_to_last_cacheable_block() helper
   (setdefault semantics, so it never clobbers a legitimately-placed
   marker).

3. Scalar whitespace: the non-list content branch
   (blocks.append({'type': 'text', 'text': str(content)})) accepted a
   truthy whitespace-only string unfiltered. Now filtered the same way
   as list-content blocks.

8/8 new tests pass in TestBlankTextBlockFiltering (including None-safety,
scalar-whitespace, and cache_control-relocation regressions on both
paths); 186/186 in the full tests/agent/test_anthropic_adapter.py file.
2026-07-24 15:54:38 -07:00
Israel Lot 6bc8d68ad7 fix(codex): scope 24h retention to Bedrock Mantle 2026-07-24 15:54:08 -07:00
Israel Lot 851b72b8f2 fix(codex): extend 24h cache retention to supported models 2026-07-24 15:54:08 -07:00
Israel Lot f54fa1bcb7 fix(codex): exclude models.github.ai from auxiliary cache retention 2026-07-24 15:54:08 -07:00
Israel Lot 48049a1d31 fix(codex): skip auxiliary cache retention on Codex backend 2026-07-24 15:54:08 -07:00
Israel Lot 339be21542 fix(codex): send prompt_cache_retention 24h for the GPT-5.5 family
OpenAI documents GPT-5.5 / GPT-5.5 Pro as extended-cache-only: in-memory
prompt cache retention is not available for them, and only
prompt_cache_retention: "24h" is supported. Responses requests that omit
the field see near-zero cached_tokens even with a stable prompt_cache_key
and identical prefixes (observed on an OpenAI-compatible Responses relay:
0 cached across repeated identical calls before; 97% cache reads after).

Send the field for the gpt-5.5 model family (bare and namespaced ids like
openai.gpt-5.5) on OpenAI-compatible Responses routes, mirrored in the
auxiliary Codex adapter, and pass it through preflight normalization.
Skipped for xAI, GitHub/Copilot, and the chatgpt.com Codex backend, which
reject or ignore body-level cache fields.
2026-07-24 15:54:08 -07:00
teknium1 98470ae33b fix(ssl): detect and repair a missing certifi cacert.pem via existing venv-repair infra
A brew Python upgrade (original report) or an interrupted venv rebuild
(v0.19.0 report in the same thread) can leave certifi importable while
its bundled cacert.pem is missing or a dangling symlink. Every TLS
connection then fails — Feishu/Telegram/WeChat/DingTalk all down —
with an opaque 'Could not find a suitable TLS CA certificate bundle'
from deep inside httpx/requests.

The existing repair infrastructure only probed
`hasattr(certifi, 'contents')`, which PASSES in exactly this failure
state, so neither the early venv self-heal nor `hermes update`'s
import-probe repair ever classified certifi as broken. Extended, not
replaced:

- hermes_cli/_early_recovery.py: the in-process probe now also
  validates that certifi.where() exists and is a plausible bundle
  (>=1KiB), so the pre-import self-heal repairs it like any other
  wiped core package.
- hermes_cli/main.py (_detect_broken_lazy_refresh_imports): the
  subprocess probe script used by `hermes update`'s venv repair
  applies the same bundle-file check inside the target venv.
- hermes_cli/doctor.py: `hermes doctor` already failed the cert
  check; `hermes doctor --fix` now repairs it (pip force-reinstall
  certifi + module-cache invalidation + re-verify), covering
  brew/manual venvs where no update marker exists. Failures funnel
  into the manual-action list with the exact command.
- agent/ssl_guard.py: the startup SSLConfigurationError hint now leads
  with `hermes doctor --fix` instead of only the raw pip command.

Fixes #29866
2026-07-24 15:53:38 -07:00
JonthanaHanh 431d2a628c fix: break unbounded 401 retry loop in credential pool OAuth path
When api_key_hint from a 401 response doesn't match any pool entry
(common with OAuth tokens where runtime_api_key rotates), the pool
rotated without marking anything exhausted and handed back a fresh
selection. Because nothing was ever marked, the pool could never reach
the "no available entries" state — the caller retried the same dead
token forever (~6 attempts/sec), starving the event loop so /stop was
never processed; only killing the gateway ended it.

Rebased onto the identity-tracking rework that landed on main
(73c4b5a045): the single-entry escape from that commit already stops
the most common OAuth case, so this fix bounds the REMAINING gap —
multi-entry pools ping-ponging A->B->A with an unmatched hint. Cap
consecutive no-mark rotations at one full lap of the available
entries, then return None so the error surfaces / fallback activates.

Deliberately does NOT mark innocent entries exhausted (the original
PR's approach): that would quarantine a healthy key for the full
cooldown TTL on a hint that provably matches nothing. No cooldown is
written by the escape, so healthy keys stay available next turn --
bounded without hammering.

The streak resets when a rotation identifies a real entry and on any
successful normal select(), so only genuinely consecutive unmatched
rotations trip the bound.

Fixes #70401
2026-07-24 15:50:36 -07:00
akb4q 835de6f764 test(compaction): byte-pin every frozen prefix generation
Hardening follow-up to the #69619 review fix. The previous regression
byte-pinned only the rescued pre-#69619 generation; older frozen entries
were covered solely by fragment assertions and a self-matching loop that
cannot detect a frozen entry mutating (the loop tests each entry against
itself).

- Pin all four _HISTORICAL_SUMMARY_PREFIXES generations as literals in
  _FROZEN_PREFIX_GENERATIONS and assert order-sensitive tuple equality
  plus detect/strip for each
- State the prepend-only contract explicitly on the tuple: never mutate
  or reorder existing entries

Negative controls verified: mutating, dropping, or reordering a frozen
entry each fail the new test, while the legacy self-matching loop still
passes under mutation — confirming the closed coverage gap.
2026-07-24 15:48:23 -07:00
akb4q 8204b27618 fix(compaction): freeze pre-change SUMMARY_PREFIX generation, restore mutated entry
Address review on #69619: the previous commit mutated the newest frozen
entry in _HISTORICAL_SUMMARY_PREFIXES and never froze the live prefix it
retired (the generation with both the four-heading discard clause and
the tools-active clause). A summary persisted immediately before
upgrading was therefore treated as an ordinary message on
resume/re-compaction, keeping the old handoff text embedded in the body.

- Prepend the exact pre-change live prefix as a new frozen entry
  (newest-first), leaving all existing frozen entries byte-identical
- Restore the Jul 2026 (#65848 class) frozen entry to its original
  four-heading text
- Pin the retired generation as a literal in
  test_summary_prefix_semantics.py so mutating or dropping it fails CI
- Make the #65848 tool-use regression position-agnostic (match the
  pre-clause generation by content, not tuple index)

Verified byte-identity of both rescued generations against the parent
commit. 233 focused prefix/resume/compressor tests pass.
2026-07-24 15:48:23 -07:00
akb4q b59cce9178 fix(compaction): strip proactive section headers from summary template
Remove three directive-heavy section headers from both the LLM
and deterministic summary templates that caused the agent to
resume stale tasks after context compression:

- Historical In-Progress State
- Historical Pending User Asks
- Historical Remaining Work

These sections read as actionable instructions even within a
REFERENCE-ONLY wrapper, hijacking the user's latest message.
The remaining sections are purely descriptive/past-tense.

Frozen prefix copies in _HISTORICAL_SUMMARY_PREFIXES updated
to match. Test 8/8 passed.
2026-07-24 15:48:23 -07:00
teknium1 306c9f7661 feat(models): add anthropic/claude-opus-5 to OpenRouter and Nous Portal catalogs
Anthropic released Claude Opus 5 (+ -fast variant) — both are live on
OpenRouter and the Nous Portal /models endpoint (verified against both
live APIs). Opus 4.8 entries are kept.

- hermes_cli/models.py: opus-5 + opus-5-fast in OPENROUTER_MODELS;
  opus-5 in _PROVIDER_MODELS[nous] (Portal serves both, curated list
  carries the base model like the rest of the Nous Anthropic block).
  Ordering: below fable-5 flagship, above opus-4.8.
- agent/model_metadata.py: claude-opus-5 -> 1M context (matches live
  OpenRouter metadata).
- agent/reasoning_timeouts.py: claude-opus-5 -> 240s stale-timeout
  floor (same as the opus-4.x thinking family).
- website/static/api/model-catalog.json: regenerated via
  scripts/build_model_catalog.py.

Both providers bill via official_models_api (live pricing), so no
_OFFICIAL_DOCS_PRICING snapshot entry is needed for these routes.
2026-07-24 13:00:15 -07:00
Victor Kyriazakos a65a647b04 fix(monitoring): correct cron operational signals 2026-07-24 18:54:46 +00:00
Victor Kyriazakos efbf2fd79c feat(monitoring): add cron operational telemetry 2026-07-24 18:54:46 +00:00
Victor Kyriazakos 9b098e7f78 fix(monitoring): enrich gateway diagnostic scope 2026-07-24 18:54:46 +00:00
Victor Kyriazakos 42bc6c8626 fix(monitoring): keep diagnostics content-free 2026-07-24 18:54:46 +00:00
Victor Kyriazakos 18b4664543 fix(monitoring): drain terminal lifecycle events 2026-07-24 18:54:45 +00:00
Victor Kyriazakos 73190cdbfe fix(monitoring): complete production OTLP runtime 2026-07-24 18:54:45 +00:00
Victor Kyriazakos 98a0260b0a fix(otel): identify Hermes resources safely 2026-07-24 18:54:45 +00:00
Victor Kyriazakos 16e774f223 fix(monitoring): harden gateway health OTLP egress 2026-07-24 18:54:45 +00:00
Victor Kyriazakos 87a15733c0 fix(monitoring): address review — persist install_id, drop dead config, single redaction path
- Cut the leftover telemetry.* DEFAULT_CONFIG block (nothing reads it) and
  the legacy telemetry-key fallback in policy.py.
- install_id: persist the minted UUID back to config.yaml on first use so
  service.instance.id survives gateway restarts (fail-open when the write
  is not possible); regression test covers the restart path.
- Remove the no-op gateway_health_export.redaction config keys. Redaction
  is always-on by design and deliberately not configurable; status output
  now says so.
- Collapse redaction to one unconditional secrets+PII scrub: drop the
  none/pii content modes (they served the dropped trajectories plane) and
  fold gateway_health.py's duplicate bearer/token/email/phone regex layer
  into agent/monitoring/redaction.py.
2026-07-24 18:54:45 +00:00
Victor Kyriazakos 505d12f662 refactor(monitoring): scope telemetry substrate to gateway health/diagnostics export
Salvages the event-spine foundation from feat/telemetry-observability
(emitter, typed events, OTLP streaming, redaction — authorship preserved in
the preceding commits) and scopes it to the plane enterprise operators need
today: gateway Service Health Monitoring plus redacted Operational
Diagnostics, exported over OTLP.

Dropped from the salvaged branch, deliberately:
- run/model/tool trajectory capture (plugins/telemetry hooks, tel_spans)
- the local JSONL + state.db tel_* store (monitoring is egress, not storage)
- usage rollups/metrics, /insights integration, bulk export
- hermes telemetry CLI (replaced by hermes monitoring status)

Those planes — shared client usage metrics and enterprise trace telemetry —
are being designed on the NeMo Relay integration with distinct consent,
policy, and export boundaries; this keeps the monitoring plane content-free
and independently enableable.

Renames agent/telemetry -> agent/monitoring, config telemetry.* ->
monitoring.*, and pins the otlp extra at OpenTelemetry 1.39.1 (matching
uv.lock; 1.30.0 conflicts with mistralai>=2.4 on opentelemetry-api).
2026-07-24 18:54:45 +00:00
emozilla edac3a7bae refactor(telemetry): cut dead schema; tests assert what's actually written
Self-review after the #51714 feedback found the reviewer's dead-table finding
was not isolated — the schema advertised far more than the code populates, and
our own tests hid it by hand-feeding fields production never sends. Make the
surface honest by subtraction.

Schema (10 tel_* tables -> 5):
  - Delete tel_gateway_events, tel_cron_events, tel_skill_events,
    tel_memory_events, tel_feedback_events — declared, never written, never read.
  - Drop columns nothing populates: tel_runs.{profile_id,estimated_cost_usd,
    cost_status}; tel_model_calls.{ttft_ms,estimated_cost_usd,cost_status,
    cost_source,end_reason,retry_count}; tel_tool_calls.{backend,retry_count,
    approval}; tel_spans.attrs_json. Cost duplicated the existing sessions
    billing columns and was always NULL here.
  - events.py / emitter _TABLE_COLUMNS / OTLP _span_attrs / rollup / preview
    display all trimmed to match.

Correctness:
  - end_reason no longer hardcodes "completed". Production finalize callers pass
    `reason` (shutdown/session_expired/session_reset); _coarse_end_reason now
    reads it and maps accordingly.
  - Fix a latent bug the trim exposed: the model_call hook passed end_reason= to
    ModelCallEvent, which the @_safe wrapper was silently swallowing — so
    tel_model_calls dropped every row in real runs. Now writes correctly.

Tests:
  - Stop hand-feeding estimated_cost_usd / turn_exit_reason that no production
    call site sends. Finalize is now driven with the real `reason` kwarg, and
    assertions cover only fields that are actually populated. This is what let
    the model_call drop hide — the suite graded on a fictional contract.

Net: a smaller system that does what it says. Verified end-to-end over the real
dispatch path (runs + connected span tree + model/tool rows populate; dead
tables gone). 160 telemetry/state/insights tests green.
2026-07-24 18:54:45 +00:00
emozilla 7c80b79bb0 feat(telemetry): write tel_spans — reconstructable run -> calls trace
Addresses the review on #51714: the trace/span layer was declared but unwired —
tel_spans was never written, call rows had no timestamp, and nothing set parent
lineage, so the store was metrics-only and couldn't reconstruct a trace.

Wire the span layer (keeping the praised star-schema shape):
  - New SpanEvent (span_id/trace_id/run_id/parent_span_id/name/kind/start_ns/end_ns)
    mapped into tel_spans via the emitter's _TABLE_COLUMNS.
  - The plugin mints a root span per run and, on each model/tool call, emits a
    SpanEvent (timing + parent = the run's root) keyed by the SAME span_id as the
    detail row, so tel_model_calls / tel_tool_calls JOIN to their span.
  - Call hooks fire on completion, so end_ns = now and start_ns is reconstructed
    from the measured latency/duration. The run's root span is emitted at finalize
    with the true run start/end.

Result: tel_spans is a connected, single-trace_id, run -> calls tree a desktop
waterfall (or any reader) can render directly, ordered by start_ns. Existing
metrics rows (tel_runs/model_calls/tool_calls) are unchanged.

OTLP: spans now flow to the exporter with their trace/parent/timing attributes.
The exporter still emits one OTel span per event rather than reconstructing OTel
SpanContexts into a connected trace tree; that projection is left for a follow-up
and the module docstring now says so plainly instead of over-claiming.

Adds test_spans_trace.py (connected-tree + detail-row JOIN) over the real dispatch
path. Accurate (pre-hook) start times, real OTLP SpanContexts, and subagent
cross-run lineage remain follow-ups.
2026-07-24 18:54:45 +00:00
emozilla 24c86e160b fix(telemetry): aggregate requires local telemetry to be on
Aggregate metrics are derived from the local tel_* tables — they're a coarsened
view of local data, not an independent capture path. With telemetry.local=false
nothing is written, so an aggregate opt-in had nothing to aggregate, yet
may_upload_aggregate() returned True and `status` showed "Aggregate metrics: on".
The config could claim a state it couldn't fulfill.

Gate aggregate on local being enabled:
  - may_upload_aggregate() now requires local_enabled AND allow_aggregate AND
    consent_state == aggregate.
  - `telemetry status` computes aggregate_enabled the same way and, when consent is
    aggregate but local is off, prints "inert: local telemetry is off — nothing to
    aggregate" instead of the opt-in hint.

Happy path is unchanged (local on + consent aggregate -> on). Adds policy and CLI
tests for the inert combo.
2026-07-24 18:54:45 +00:00
emozilla 72b7af73bd refactor(telemetry): drop "plane" terminology
Rename the telemetry tiers away from the borrowed control-plane/data-plane
jargon to plain language, across code, CLI output, config, and docs:

  - "local plane"        -> "local telemetry"
  - "aggregate plane"    -> "aggregate metrics"
  - "trajectories plane" -> "trajectories" / "telemetry.trajectories"
  - "three planes with a hard wall" -> "three settings, isolated from each other"

User-facing `hermes telemetry status` now reads "Local telemetry: on" /
"Aggregate metrics: off" / "Content export: off (trajectories disabled)".
The OTLP resource attribute key telemetry.plane is renamed to telemetry.scope
(wire-level identifier; nothing consumes it yet).

No behavior change — wording only. Status renders identically apart from the
labels; tests updated to match the new strings.
2026-07-24 18:54:45 +00:00
emozilla e829d03fec refactor(telemetry): drop policy.resolve(); read config directly
policy.resolve() / TelemetryDecision was a read-only projection used only by
`hermes telemetry status` for display. The actual behavior gates already read
telemetry.* straight from config: the emitter (whether to write) and the plugin
loader (whether to auto-load) each call .get("local", True) on the loaded config,
never through policy.

Make config the single chokepoint the status command reads too: it now resolves
local/allow_aggregate/consent_state inline from the loaded config, the same way
the other gates do. policy.py keeps only what config can't express on its own —
the consent constants, ensure_install_id(), and may_upload_aggregate(config) as a
pure function (the gate a future uploader must consult). resolve() and the
TelemetryDecision dataclass are removed; policy.py drops 107 -> 70 lines.

No behavior change: status renders identically, and the default-on local plane is
still defaulted in DEFAULT_CONFIG plus a fail-safe .get(..., True) at each gate.
2026-07-24 18:54:45 +00:00
emozilla 3e28eaccde feat(telemetry): local-first telemetry & observability
Add a built-in telemetry system that records what the agent does — workflows,
model calls, tool calls, errors — to the local machine, powers `/insights`, and
can export to an operator-chosen destination. Default-on locally; nothing leaves
the machine unless the user exports it or opts into the aggregate plane.

Three planes with a hard wall between them:
  - local: full-fidelity observability (real model/provider/tool names), on by
    default, never leaves the machine.
  - aggregate: opt-in metadata, default off. No uploader ships — consent is
    recorded via telemetry.consent_state, and `preview` shows what would be
    produced, computed locally.
  - trajectories: full message content, opt-in, exported only to the operator's
    own destination.

Mechanism:
  - Bundled `telemetry` plugin registers observational lifecycle hooks
    (on_session_start / post_api_request / post_tool_call / on_session_finalize).
    No core call sites are edited; hooks already carry the data.
  - Fire-and-forget emitter: emit() returns in microseconds, never blocks or
    raises into a model/tool call. A daemon thread writes events to an
    append-only JSONL log and the tel_* tables in state.db (its own sqlite
    connection, separate from SessionDB).
  - tel_runs / tel_model_calls / tel_tool_calls live in the declarative
    SCHEMA_SQL and are reconciled automatically; SCHEMA_VERSION 16 -> 17.
  - metrics derives rollups for /usage and /insights; rollup builds per-run
    summaries for `hermes telemetry preview`.

Consent is config, not a parallel command surface. The config file is the root
of trust: set telemetry.consent_state with `hermes config set`, or pin any
telemetry.* key (including allow_aggregate) via managed scope, which overrides
the user's value per key. `hermes telemetry` exposes only what config cannot:
status (report), preview (query), and export.

Export:
  - exporter_bulk writes telemetry (and, when the trajectories plane is enabled,
    session content) to ndjson/json.
  - otlp_exporter streams spans to a configured OpenTelemetry Collector over
    OTLP/HTTP. The SDK is an optional extra (hermes-agent[otlp]), lazily
    installed via tools.lazy_deps on first use.
  - Secrets are always redacted on every export path
    (redact_sensitive_text(force=True)); content export is gated by the
    trajectories plane, and PII scrubbing follows telemetry.content_redaction.
    OTLP auth headers reference environment variable names, never inline values.

No outbound emission to Nous. The aggregate uploader is intentionally not built.
2026-07-24 18:54:45 +00:00
teknium1 d4b867cf9f fix(windows): sweep remaining unguarded text-mode subprocess sites codebase-wide
AST-driven pass over every subprocess.run/Popen/check_output/check_call/call
with text=True (or universal_newlines=True) and no explicit encoding=:
append encoding='utf-8', errors='replace' at the kwarg site. 136 call
sites across 28 files (cli.py, hermes_cli/main.py, tools_config.py,
environments, computer_use, gateway, scripts, skills helpers, agent/*).

Together with the salvaged #55339/#60741 commits this closes out issue
#53428's bug class; the salvaged #60751 linter rule in
check-windows-footguns.py now enforces it repo-wide (verified: 807 files
scanned, zero findings).
2026-07-24 11:45:57 -07:00
Stoltemberg c89481db5e fix: add explicit UTF-8 encoding to all subprocess text=True calls (#53428)
On Windows with Chinese locale (GBK), subprocess.run(text=True) without
explicit encoding causes UnicodeDecodeError crashes. This fix adds
encoding='utf-8', errors='replace' to all subprocess.run() and
subprocess.Popen() calls that use text=True across 76 non-test Python files.

Fixes #53428 (master tracker for Windows GBK locale crash).

Note: credential_pool.py and electron changes excluded per reviewer request —
those will be submitted as separate focused PRs.
2026-07-24 11:45:57 -07:00
xxxigm a9e7b32162 fix(fallback): allow xai-oauth → xai failover on shared host/model
Base-url+model dedup was meant for custom shim aliases, but it also
skipped first-class providers that share an inference host while using
different credentials. That stranded xai-oauth spending-limit failover
to the xai API-key provider when both used the same model slug.
2026-07-24 23:22:11 +05:30
kshitij c769081803 refactor: extract sync_credential_pool_entry_id helper
Replace 3 duplicated entry_id resolution blocks (try/except +
entry_id_for_api_key + fallback to None) in agent_init.py,
chat_completion_helpers.py, and switch_model with a single
sync_credential_pool_entry_id(agent) function in agent_runtime_helpers.

Follow-up to #70323.
2026-07-24 22:58:47 +05:30
Gille 73c4b5a045 fix(auth): stop stale-key credential recovery loops
Track the selected credential by stable pool entry ID so token refreshes and shared cursor movement cannot detach failures from the entry that issued them. Stop unmatched single-entry pools from reporting a no-op rotation as successful recovery.

Co-authored-by: Maxim Esipov <maksesipov@gmail.com>
2026-07-24 22:58:47 +05:30
Brooklyn Nicholson a96f7c805b feat(i18n): add Arabic (ar) catalog for agent/CLI messages
Registers `ar` in the supported-language set and alias table and ships
locales/ar.yaml at full key and placeholder parity with en.yaml, covering
approval prompts and gateway slash-command replies. Identifiers, commands,
paths, config keys, model/provider names, and {placeholder} tokens are kept
verbatim.

Co-authored-by: Da7-Tech <286182457+Da7-Tech@users.noreply.github.com>
2026-07-24 12:10:03 -05:00
teknium1 397e9fc1e4 Reapply "Merge pull request #30179 from NousResearch/feat/iron-proxy"
This reverts commit c6dc7c03c3.
2026-07-24 09:49:00 -07:00
brooklyn! 80e575dfba Merge pull request #70604 from NousResearch/bb/profile-routing-super
fix(sessions): keep a conversation on its owning profile through branch and compression
2026-07-24 10:11:10 -05:00
flyingdoubleg e9a7c18890 fix(memory): honor disabled toolsets for provider tools 2026-07-24 13:00:53 +05:30
Brooklyn Nicholson e9a243ef78 fix(state): inherit and stamp profile_name across rotation and branch children
profile_name was only written on the agent's initial lazy create
(e8b7ce8c1); every parented child row — compression rotation, TUI
/branch, desktop branch first-persist — was created without it. A
non-default profile's lineage therefore turned NULL on its first
compression or branch and aggregated as "default" in unified session
lists, completing the cross-profile session-jump.

Fix the class at the DB layer: _insert_session_row's parent backfill now
COALESCEs profile_name from the parent alongside cwd/git_* (#64709
pattern), so any parented child inherits its lineage's owning profile.
Stamp it explicitly at the three create sites as well — compression
rotation (mirroring _ensure_db_session), TUI session.branch, and the
TUI first-prompt row persist — so rows are self-describing even when the
parent row predates the profile_name column.
2026-07-24 01:49:22 -05:00
Teknium 23476207bc feat(moa): default advisor fanout to user_turn — the cheapest cadence
Flips the default fan-out cadence from per_iteration (advisors re-run on
every tool iteration, multiplying advisor spend by tool-loop depth) to
user_turn (advisors run once on the first message of each user turn; the
acting aggregator works the rest of the tool loop with that turn's
advice). Until per-mode benchmarks justify a costlier default, MoA
defaults to the cheapest, lowest-impact cadence (#67199).

One default for everyone — no split legacy/new-preset semantics; presets
that want per-step advising set fanout: per_iteration explicitly. All
three modes (user_turn / per_iteration / every_n:N) remain selectable;
every_n:1 still collapses to per_iteration (semantic identity), while
unparseable values now fall to user_turn (the default).

Docs updated with a default-change note; the per-iteration rerun test
pins its mode explicitly.

Co-authored-by: skyer-flyyy <188930297+skyer-flyyy@users.noreply.github.com>
2026-07-23 21:07:18 -07:00
Alan Harman-Box bc744d30e0 docs: document 57-char system prompt truncation in authoring guide and curator
The skill-authoring guide and curator prompt both reference
descriptions as the primary discovery mechanism but never mentioned
the 57-char system prompt truncation. Add explicit guidance:

- Authoring guide: frontmatter docs, template comment, size limits,
  pitfall #3 with good/bad examples, verification checklist
- Curator prompt: parenthetical noting the 57-char window when
  writing umbrella skill descriptions
2026-07-23 21:06:56 -07:00
Alan Harman-Box accbf4d912 feat: show system_prompt_preview when skill description exceeds prompt limit
When a skill is created or edited with a description longer than
SKILL_PROMPT_DESC_LIMIT (60 chars), the tool response now includes a
system_prompt_preview field showing exactly what the system prompt
skill index will display. This gives the agent immediate feedback to
self-correct truncated trigger phrases.

Also adds tool schema guidance about the 57-char window and fixes a
stale docstring in skill_commands.py that incorrectly claimed the
system prompt renders the full description.
2026-07-23 21:06:56 -07:00
Alan Harman-Box 5eb772111d refactor: extract SKILL_PROMPT_DESC_LIMIT constant and normalize description helpers
The system prompt skill index truncates long descriptions to 57 chars,
but this limit was a hardcoded magic number. Extract it as a named
constant and factor the normalization logic into a shared private
helper so the extraction function and the new truncation predicate
cannot drift.

No behaviour change — pure refactor.
2026-07-23 21:06:56 -07:00
Teknium 13fe08d7e9 fix(context-engine): short-circuit the inherited no-op select_context before any per-request work
Verification follow-up for the #51226 salvage: the host call site guarded
select_context with hasattr(), but the ABC defines a default on every
engine, so the built-in ContextCompressor (and any non-implementing
engine) still paid per-request shallow copies of the conversation
history plus a hook call on every provider request. Identity-check the
bound method against ContextEngine.select_context and return the
request untouched — mirroring the existing base-method short-circuit in
_notify_context_engine_turn_complete — so the default path does zero
work, not just produces an identical result.

Adds two pins: the base no-op is never invoked (patched-to-raise base
stays silent), and ContextCompressor.__dict__ contains neither new verb.

Also registers the contributor email mapping for @chaos-xxl.
2026-07-23 19:44:35 -07:00
xue xinglong 5f65f0b0f8 fix(context-engine): snapshot select_context read-only inputs; scope on_turn_complete coverage doc
Addresses the hermes-sweeper review on #51226:
- _apply_context_engine_selection now passes shallow copies of the read-only
  conversation_messages / incoming_message to the hook, so an engine mutating
  them in place cannot corrupt persisted transcript state (enforces the
  request-only contract, not just documents it). Adds a mutation-regression
  test asserting persisted history + incoming message are untouched.
- on_turn_complete docstring: scope the coverage claim to the standard
  finalization seam. Some abnormal early-return paths (content-policy block,
  provider terminal failure) currently persist+return without finalization and
  don't emit the hook; documented as best-effort with a shared-seam follow-up,
  rather than over-promising a guaranteed callback for every early exit.
2026-07-23 19:44:35 -07:00