Persist paused state, timestamp, reason and no first trigger in the original
locked creation write. Forward the same boolean contract across CLI, tool,
gateway API and dashboard API, validating at the store boundary. Preserve
explicit operator force-run behavior and normal enabled creation.
The live CLI probe also caught the command shim dropping failure return codes;
forward them so invalid creation reports exit 1 rather than success.
Credit earlier atomic-creation work in #78935 and #94952 and the focused
implementation in #104578. The broader manifest staging layer is not imported.
Co-authored-by: Konstantin Khlopkov <konstantin.khlopkov93@gmail.com>
Co-authored-by: Chloé DuPont <321112755+misschloedupont@users.noreply.github.com>
Salvage the unit-budget implementation from #104745, replacing its test
matrix with two invariant tests and covering the sibling graceful start.
Keep unprivileged property reads, finite fallbacks, real manager errors,
and post-restart health verification.
Native disposable user unit: old client timed out after 15.03 seconds;
new client completed the same 16-second stop transaction in 16.13 seconds.
The unit stayed active with a new PID; missing-unit errors stayed errors.
Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
Reuse the effective gateway config and shared session platform policy before hashing model-facing metadata. Preserve original routing state and cover enabled/disabled redaction across all busy injection routes.
Address independent review: include existing alternate and parent routing fields, and distinguish event message ID from source message ID without inferring a reply target. Expanded live probe first failed on parent_chat_id, then passed with the additional fields.
Trim salvage of #104047: preserve the shared gateway injection boundary, but keep lossless JSON entirely per message. Do not change the system prompt, steer ABI, or synthesize a delivery target. Live handler-to-agent A/B verified; serialized directory tests and independent review pending.
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
The corrupt-cause recovery guidance hardcoded `~/.hermes/backups/` while
every other path in the same message follows the active HERMES_HOME
(`{db_path}` is already interpolated). A custom-home or named-profile
deployment was told to restore from a directory that may not exist at all,
mid data-loss incident. Both sites (turn-completion explainer and gateway
startup broadcast) now interpolate `<hermes_root>/backups` via
get_default_hermes_root(), matching hermes_cli/backup.py's real backup
location.
Fixes#104250
Salvage only the demonstrated delivery and session-header fixes. Leave queue admission, other CORS expansion and unrelated optimizations out of this bug pass.
Co-authored-by: DevvGwardo <25094504+DevvGwardo@users.noreply.github.com>
Co-authored-by: Frowtek <frowte3k@gmail.com>
Resolve task sources and committed member messages from the durable log,
not disposable policy projections. Record late outcome receipts without
reactivating settled discussions; retain frozen prompts and retry behavior.
Slim redo of the durable-log and late-publication portions of #104020.
Unlike that proposal, retain retries and do not hide reconstruction errors.
Also retain the source after the bounded thread transcript ages it out.
Refs #104007
Co-authored-by: Halldrix <12357213+Halldrix@users.noreply.github.com>
The redelivery hook keyed on the canonical flood_control:<seconds> result, so
two real refusals slipped past it and armed no timer, leaving the reply for the
next restart. A short wait that outlived the send retries raised instead of
failing closed, and an edit refused again after its inline wait returned the
platform's raw text. Both now fail closed canonically, the second carrying the
new delay rather than the first refusal's.
The ledger also accepts a row still carrying the platform's own wording, so a
row persisted by an unnormalized path is dated from the delay it states instead
of the generic default. Without that a boot sweep claims it at once and spends
its one attempt inside the penalty. Matching requires the flood wording as well
as a delay, so an unrelated retry suggestion is never read as a flood.
Six new assertions fail without this change. 770 passed across the ledger,
Telegram, send-retry and queued suites.
Round-2 review. Making _arm_flood_timers_for_waiting_rows read the ledger in a
worker thread introduced a race: a flood refusal recorded during that thread's
yield had its own _schedule_flood_redelivery request declined (the timer still
held the slot), and the timer then cleared its slot on a snapshot taken before
the new row committed, leaving that reply with no timer until the next reconnect
or restart. The read is a single indexed SELECT; doing it synchronously keeps the
arm decision atomic with respect to concurrent schedule requests. A test drives a
refusal during the timer's own redelivery send and asserts the same timer arms it.
Second review pass on the flood-retry change.
- sweep_recoverable returns a dead owner's not-yet-due flood row flagged
`adopted` (with its `not_before`) instead of dropping it from the result,
so _claim_pending_obligations clears its session's resume_pending flag like
every other claimed row; the answer is in the ledger and the turn must not
be re-run at boot. _redeliver_claimed_obligations skips adopted rows and
still arms the timer for them.
- A legacy row without adapter_profile is normalised to 'default' when the
boot sweep claims or adopts it: the runtime sweep matches profiles exactly
and the timer asks for 'default', so an adopted NULL row could only wake the
timer without ever being sent.
- A sleeping timer is replaced by a shorter refusal's timer; the shorter one
re-arms for the longer row after its sweep. A timer already sweeping is
never cancelled from outside.
- pending_flood_retries is read in a worker thread like the other ledger calls.
- Adoption tests seed a distinct dead-owner stamp and assert ownership moves.
- Scratch tags and long comment lines removed.
Review on the first cut reproduced two gaps.
The raw UTF-16 length of the stored reply said nothing about how many
requests the adapter made: MarkdownV2 escaping turns 3000 dots into 6000
units, two Telegram messages, and the send result does not say which chunk
the platform refused. Drop the length-based certainty (and its helper and
constant) and mark every flood redelivery with the rate-limit marker, at
runtime and at boot. The cost is a marker on a reply nothing of which was
delivered; the alternative was a silent duplicate.
Releasing an unsent runtime claim always wrote send_path_degraded, so a flood
row whose resume flag could not be cleared (or whose adapter vanished before
dispatch) left the flood timer's list and stayed stranded until a reconnect or
restart. The claimed row now carries its pre-claim error and the release
writes it back, so the row waits the platform's figure once more and is sent
on the next timer.
The Telegram adapter fails a flood-controlled final send closed as
flood_control:<seconds> so the send coroutine never sleeps a long penalty
(#91969), on the understanding that the delivery ledger owns the wait. The
ledger did not: sweep_failed_for_runtime only replayed send_path_degraded
rows, so a flood-refused row sat in 'failed' until the next restart, whose
sweep_recoverable then redelivered it hours late prefixed with "Recovered
reply, the gateway restarted during delivery, so this may be a duplicate".
Observed: a reply refused at 08:58 UTC (both the MarkdownV2 send and the
plain fallback got flood_control:185) arrived at 12:15 UTC after a restart,
labelled as a possible duplicate although the platform had never accepted it.
Ledger (gateway/delivery_ledger.py):
- flood_control rows are runtime-retryable, but only once their own deadline
has passed: the refusal's updated_at plus the platform's wait
(flood_not_before). Neither an early timer nor a reconnect sweep spends a
redelivery attempt inside the penalty window.
- A flood refusal of a reply that fits in one Telegram message (4096 UTF-16
units) proves non-delivery, so that redelivery carries no duplicate marker.
A chunked reply may have had its first chunk accepted before the refusal
and keeps the marker.
- Claiming a flood row clears the stale refusal (last_error NULL, state
'attempting'), so a resend interrupted before mark_delivered is seen as
uncertain by the next boot and gets the marker.
- At boot, a dead owner's flood row that is not yet due is adopted (owner
re-stamped, no attempt spent) instead of being resent early.
- pending_flood_retries() lists this process's waiting flood rows per adapter
identity with the earliest deadline.
Runner (gateway/run_startup.py):
- _schedule_flood_redelivery arms one timer per adapter identity that runs
the existing runtime sweep after the wait (capped at 15 minutes per sleep;
the row's deadline, not the timer, decides eligibility, so a capped timer
wakes early, sends nothing, and re-arms for the remainder). The slot stays
occupied until the timer ends and only the running timer may arm its
successor into it, so a refusal during the sweep can never strand the row.
- _arm_flood_timers_for_waiting_rows runs after every redelivery pass (boot
and runtime), covering adopted rows, rows skipped as not yet due, and rows
refused again.
Adapter (gateway/platforms/base.py): _finalize_delivery_obligation arms the
timer on a flood_control failure, best-effort inside the existing try.
Tests: 35 in tests/gateway/test_delivery_ledger_flood_retry.py, with a
controllable clock shared by ledger and runner. Each guard was mutation
tested. Existing ledger, reconnect and redelivery suites (131 tests) pass.
Keep parsing, contracts, gates and persisted goal mutations in one dispatcher. Adapters retain authorization, rendering and scheduling; TUI drafting resolves the target session profile off the RPC reader. Document ACP as unsupported rather than implying a goal loop exists.
A flat per-image constant (1500 in the trigger estimator, 1600 in the tail-budget walk) is wrong in
both directions: a screenshot costs ~1,100 tokens on one provider and 4,000+ on a local mmproj
model. In a GUI loop on a 64K window the estimate sat at ~20K while the real prompt passed 80K,
so compaction never fired and the provider rejected every request (#70328).
The provider prices every image exactly on the request that carries it, so the cost is
observable from usage alone, with no vendor formula: with a fresh usage anchor, the residual
between the next real prompt_tokens and anchor + text-only delta is the price of the N images
that delta introduced.
- agent/image_token_cost.py: calibrate_from_usage() runs in record_response_usage before the new
anchor is captured; the learned value (EMA, plausibility-banded) is kept per model@host in
~/.hermes/cache/image_token_costs.json and bound per turn through a ContextVar.
- estimate_messages_tokens_rough, _content_length_for_budget (tail walk) and gateway hygiene all
read the same bound value, so trigger and walk agree; the per-message memo now caches text
tokens and image COUNT so a recalibration re-prices cached rows.
- One flat default (1500) remains only until the first vision turn; the duplicate 1600 is gone.
evals/token_accounting/ab_image_cost_calibration.py (real AIAgent, fake provider pricing images
at 4,000, one screenshot per turn, 64K window): main learns nothing (1500) and the tail walk
under-prices its own protected tail by 56.5%; this branch learns 4,374 after one vision turn
and the walk's error is +8.5%.
Reporter and first-fix credit: @JonthanaHanh (#70328, #70463).
Two parallel "real usage" mechanisms fought each other: the usage anchor (real + delta) and the
compressor's rough/real projection (should_defer_preflight_to_real_usage with
last_rough_tokens_when_real_prompt_fit / _pending_request_rough_tokens / note_request_rough_estimate
baselines). The projection stored an anchored, real-scale figure as its "rough" baseline, so a
rewind that invalidated the anchor produced phantom growth and a spurious compaction (#103391).
Now there is one authority:
- Post-tool gate (turn_preflight.compress_after_tool_results): anchored figure first (the raw
last_prompt_tokens ignored the tool results just appended), then real, then rough.
- Gateway hygiene (run_turn._hmwa_hygiene_plan): real session count, else the anchor persisted on
the session row, else rough.
- Preflight / pre-API gates: an anchored figure is never deferred. A whole-context rough estimate
over threshold waits ONE request for the provider's real count instead of compressing on a guess
(first request, rewind/edit-resend, reloaded history without a persisted anchor).
- The wait is one request, never a disable: a provider that omits usage
(note_usage_less_response, #2153 class), a real reading already over threshold, a rough figure
past the whole window, and provider-proven overflow all compress immediately; the post-compaction
latch (#36718 / #104192) is unchanged.
- Projection baselines and their bookkeeping deleted (-101 LOC in context_compressor); the fixtures
that scripted whole-history estimates now state the fact they relied on (provider omits usage).
Fixes#103391 (closes#103397 by construction — the baseline it repaired no longer exists).
feat(delegation): a failed child of a running detached fan-out is surfaced immediately, as a non-durable notice (66 min dead wave; review-found final-result loss closed)
- _exit_with_failure_verdict / _resolve_gateway_exit_verdict move out of the
run.py facade into run_shutdown.py, which already owns _restart_via_service
and GATEWAY_SERVICE_RESTART_EXIT_CODE (no alias import needed).
- The running-shutdown tail keeps its early `return False` on a failure verdict,
as on main, so a failure exit does not first drain cron/MCP; the helper still
re-checks it for the startup-abort path.
- The three start_gateway tests share one _patch_aborted_startup helper.
Adapt the shared exit-verdict resolver and service-restart fallback from JoaoMarcos44’s implementation in f77586ef7683c17d5890e63e38604a68d5ea11ce (#103248) for the earlier #103208 branch.
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
The #102827 corruption is pure zero holes -- frames lost across a WAL
generation. SessionDB.close() produces exactly that when it runs against a
file another live handle is still writing: PRAGMA wal_checkpoint(PASSIVE),
then the connection close that lets SQLite unlink -wal/-shm. The dangerous
event is a physical close overlapping any other live physical lifetime for
the same path, so both sides of it are closed here.
Late write vs. close: a cron watchdog timeout only stops waiting, and
ThreadPoolExecutor.shutdown(wait=False) cannot interrupt a worker already
inside run_conversation. The agent and its registry reference are now held
until that worker's Future completes, so its last frames land before any
checkpoint.
Close vs. open: the per-path barrier now COUNTS admitted teardowns. A path
can own several closes at once -- the current generation's final release and
a retired generation's drain are admitted independently under the registry
lock, and the per-path mutex only serializes teardowns that already entered
it. With one bare event per path, a releasing thread descheduled between
generation removal and the mutex let the next teardown to settle remove and
signal the shared event: close_all() returned over a pending close and
acquire() published a replacement writer on top of a handle still inside
checkpoint/unlink. _TeardownBarrier tracks event + pending count,
_admit_teardown_locked registers each close in the same lock section that
removes the generation, and only the last settled teardown lifts the
barrier. Physical I/O stays outside the registry lock and unrelated paths
still progress independently.
The auto-archive sweep called release_or_close in its finally while the
import was local to a different function, so every eligible sweep raised
NameError, the outer except Exception swallowed it at debug level, and the
borrowed registry reference was never returned -- a holder leak that pins a
retired generation open. The helper is now bound in the calling scope.
Remaining in-process writable SessionDB() call sites (trace upload, the
API-server profile cache, the web-server writable paths, startup schema
reconcile) go through the canonical registry acquire/release_or_close, and
gateway maintenance borrows pinned handles instead of iterating an unpinned
snapshot.
Regressions: overlapping final releases of the current and retired
generations in both orderings with the first paused before the lifecycle
mutex, teardown-error settlement, an unrelated-path control, and refcount
assertions for the auto-archive sweep on success, on failure, across
repeated sweeps and with auto-archive disabled.
Fixes#102827
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzxCWw6SuHXhXMdkiEwMa2
A rate-limited error classifies as network above and the retry loop only breaks
on a non-transient, non-rate-limited error, so the fallback-path guard could
never fire. Tests: the classifier delegator's vocabulary tests were a
change-detector; two rate-limit loop tests were subsumed by the exhaustion
contract (3 sends, no notice, no fallback).
Review follow-up on the salvaged #100072 flood handling. Honouring retry_after
is right, but sleeping it verbatim in _send_with_retry would pin the send
coroutine for a long penalty (a 97-minute FloodWait once froze inbound on
every platform, #91969 -- the Telegram adapter already fails closed at 5s for
the same reason and hands the wait to this loop). Past 60s the typed failure
goes back to the delivery ledger, which owns redelivery after the cooldown.
The six classifier tests collapse into one parametrized contract.
Flood-capped / rate-limited sends (Telegram FloodWait, Weixin bare RuntimeError)
were not honored as retryable when retry_after was absent, so _send_with_retry
fell through to the truncating plain-text fallback, which re-entered the server
ban and dropped the message tail — silently burning the turn with no delivery.
Three changes to _send_with_retry:
1. Route rate-limit checks through _is_rate_limited_error (single wrapper over
classify_send_error) so rate limits are treated as transient even when the
platform omits retry_after.
2. Reclassify rate-limit per retry attempt, not just the initial send, so the
in-loop break/continue reflects the CURRENT attempt (covers transient->flood
and rate-limited->formatting transitions).
3. Never take the truncating plain-text fallback for a rate-limited send — return
the typed failure so the delivery ledger owns redelivery after the cooldown.
And when retries exhaust on a rate-limited / retry_after-carrying failure,
return BEFORE sending the delivery-failure notice: the notice send would land
inside the same flood penalty and re-enter the ban ([0, 189, 378, 378] -> 3
sends, no 4th). Ordinary exhausted network errors keep the existing notice.
Honors server retry_after (when present) as the backoff delay instead of the
default exponential schedule.
Rebased onto current main (was 5163 commits behind, CONFLICTING) and folded in
reviewer feedback. Regression coverage in tests/gateway/test_send_retry.py:
rate-limit classifier, rate-limited-without-retry_after retries & succeeds,
rate-limited exhaustion returns a typed failure with no fallback and no notice
inside the active flood penalty, and failure-kind transitions between attempts.
Follow-up to the truncation removal: with the full reply text in the prepared message, a
`@file:`/`@url:` reference inside the *quoted* message reached the context-reference expander
and read a local file on the replier's behalf (the old 500-char slice hid this for long
quotes; short quotes were already exposed on main). Expansion now runs on the new message
text only; the reply pointer is prepended afterwards.
GitLab (lib/gitlab/web_hooks.rb, app/services/web_hook_service.rb) sends
webhook-id and webhook-timestamp on EVERY delivery and adds webhook-signature
only when a signing token is configured. Selecting the HMAC path on any
webhook-* header would 401 every legacy X-Gitlab-Token install the moment it
upgrades to GitLab 19, so only the signature header selects the path; id and
timestamp then travel with it and the validator still fails closed when either
is missing. svix-* keeps its existing any-header selection.
Tests trimmed to the invariant bar: one parameterized contract (whsec_ and raw
secrets accept; wrong secret, tampered body and stale timestamp reject) plus the
GitLab legacy-token coexistence contract. evals/webhook_auth/standard_webhooks_ab.py
drives a real aiohttp WebhookAdapter on a dedicated loopback port with real
signed requests for before/after evidence.
Related: #47849 (HwangJohn, cherry-picked here), #92024 (earlier salvage of
#47849), #102080 and #103167 (same alias fix, same target).
The salvaged helper was appended to the gateway/run.py facade; new behaviour
belongs in a topical sibling. agent/session_activity.py already owns the
activity-snapshot contract the three render sites read from, so the formatter
moves there as format_iteration_progress and the three call sites import it at
module level instead of late-importing the facade.
Tests trimmed to the salvage bar (<= 2 invariant tests per fix): one
parameterized contract on the helper (unbounded/None hide the ceiling, a real
budget keeps N/M) plus the contributor's end-to-end busy-ack test through the
real render path. The per-site heartbeat/timeout tests exercised the same
helper through mocks and are dropped; evals/gateway_status_render/
iteration_ceiling_ab.py drives all three real render sites with a real AIAgent
for before/after evidence (3 sentinel leaks on main -> 0).
Related: #103109 (same fix, same target module; credited in the PR),
#102845 (third render site, closed by its author in favour of #102817).
Teknium