startSocket() awaits useMultiFileAuthState() and fetchLatestBaileysVersion()
before it creates a socket or registers event handlers, and the close handler
re-entered it via a bare setTimeout(startSocket, ...). That leaves two
unrecoverable failure modes on a reconnect:
- a rejection is an unhandled promise rejection (fatal on modern Node)
- a hang leaves the bridge permanently disconnected with nothing left to
retry, while its HTTP server keeps answering 503 to the gateway
The second mode was observed in the field: fetchLatestBaileysVersion() is a
plain fetch to raw.githubusercontent.com with no AbortSignal, and after a
stream:error 503 disconnect the bridge logged 'Reconnecting in 3s...' once
and then sat silent and disconnected for 27+ hours until manually restarted.
Fix, as two pure helpers in bridge_helpers.js (keeping bridge.js side-effect
free to test):
- createReconnectScheduler(): every (re)connect entry point now catches a
failed startSocket() and reschedules it instead of dying or going silent
- createVersionResolver(): bounds the version fetch with a 15s timeout and
falls back to the last known-good version (or the Baileys default before
first success) instead of pending forever
Follow-up fixes for the combined Matrix crypto salvage (#71073,
#71543, #71547):
1. Construct _pickle_key from client.device_id (resolved from whoami)
instead of self._device_id (the configured value). Without this, when
#71543 makes the token's real device win over a stale
MATRIX_DEVICE_ID, the pickle key is built from the stale value and
the Olm account is stored under a key that can never be looked up
again — perpetuating the same decryption failure the PRs aim to fix.
2. Skip _migrate_legacy_crypto_pickle when the store was just deleted
by _reset_crypto_store_if_device_changed — there is no account to
migrate. Also check the migration return value and log a warning on
failure instead of proceeding to olm.load() which fails with a
cryptic BAD_ACCOUNT_KEY.
3. Add a local dict cache (_enc_info_cache) to _CryptoStateStore so the
homeserver fallback in get_encryption_info() does not make a network
round-trip on every is_encrypted() call. MemoryStateStore does not
implement set_encryption_info, so the existing cache-back is a no-op.
4. Log homeserver encryption-info query failures at DEBUG level instead
of silently returning None (which would cause OlmMachine to treat an
encrypted room as unencrypted).
5. Update _CryptoStateStore docstring to mention the homeserver fallback.
The account was written under the new pickle key before sessions were
re-pickled. The account is effectively the migration's commit marker —
once it reads under the current key, the fast path short-circuits every
later startup — so a sweep that errored or was interrupted left the
remaining legacy-key sessions stranded permanently with no retry.
Sweep first, commit the account last, and return False on sweep failure
so the migration is retried on the next start.
Also corrects the unreadable-row log: it claimed rows were being dropped
while no DELETE was ever issued. Such rows are left in place (already
unusable; deleting crypto material on a guess is not worth it) and the
message now says so.
Adds session-sweep coverage, which was previously absent: rows rewritten
under the current key, rows already current left alone, unreadable rows
left in place, and a failed sweep that leaves the account uncommitted.
The existing migration test now fakes the olm C-extension so the suite
no longer requires libolm.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Olm account pickle key is derived from the account ID plus the
configured device ID (acct:device_id). If the crypto store's account was
created before MATRIX_DEVICE_ID was set — e.g. the very first password
login, where the device ID is only known after connecting — it gets
pickled under "<acct>:default". Setting MATRIX_DEVICE_ID afterwards (a
reasonable thing to do once you know the device ID you want to pin)
changes the derived pickle key, and every subsequent unpickle attempt
fails with BAD_ACCOUNT_KEY. In optional-E2EE mode that failure is
swallowed and encryption silently stays disabled instead of surfacing an
actionable error.
_migrate_legacy_crypto_pickle() detects the BAD_ACCOUNT_KEY failure,
tries the known legacy pickle keys, and re-pickles the account (plus
every stored olm/megolm session — sessions share the same pickle key, so
migrating only the account would leave them unreadable on the next
decrypt and silently break key sharing with peers) under the current
key. It only reports failure when no known key can unpickle the
account, with a log message pointing at what changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
connect() resolved client.device_id as `self._device_id or resolved_device_id`,
so a configured MATRIX_DEVICE_ID masked the live whoami() device. With
persisted A, configured A and a rotated token reporting B, the reset
compared A to A and never fired — exactly the token-rotation case this PR
claims to handle.
An access token is bound to one device and the homeserver only accepts key
uploads for that device, so a configured value naming a different one
cannot work. The live whoami() device now wins on conflict and logs an
error naming both. The configured value is still preferred when whoami()
reports no device.
Adds a connect()-level regression for persisted A + configured A +
whoami B, and corrects test_connect_uses_configured_device_id_over_whoami,
whose stated premise this inverts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The crypto store is keyed by Matrix user ID, not device ID, so swapping in
a new access token (which mints a new device_id) silently inherits the
previous device's Olm account. That account's identity keys can never be
published under the new device ID, and the pickle key embeds the old
device ID anyway — the result is stale-key mismatches and cross-signing
signatures the homeserver refuses to replace, degrading E2EE in ways that
are hard to diagnose (peers silently withhold room keys).
_reset_crypto_store_if_device_changed() compares the store's persisted
device ID against the live one at connect time and wipes the store on
mismatch, so a fresh Olm account is generated for the new device instead
of reusing stale key material.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_CryptoStateStore.get_encryption_info only consulted mautrix's in-memory
MemoryStateStore, which has no record of m.room.encryption for rooms the
bot joined in the past (the raw-sync path never feeds those state events
through set_encryption_info). On a fresh crypto store this returns None
for all previously-joined rooms, so OlmMachine reports them as unencrypted,
never tracks peer devices, and silently drops all inbound messages.
Fix: pass the mautrix Client into _CryptoStateStore so get_encryption_info
can fall back to a live GET /_matrix/client/v3/rooms/{room_id}/state/
m.room.encryption query when the in-memory store returns None. The result
is cached back via set_encryption_info so subsequent lookups (and
OlmMachine device tracking) hit the fast path.
Post-tool compression path passed context_compressor.last_prompt_tokens (0 in the no-usage
fallback) to _compress_context instead of the overhead-aware _real_tokens computed just above
— same tool-blind bug as the overflow handlers (upstream PR #77169 review, teknium1). Also adds
production-path regression tests asserting the 413, context-overflow (two wordings), and
Anthropic long-context recovery handlers pass estimate_request_tokens_rough(..., tools=...)
(sentinel-patched) to _compress_context.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root fix (Option A) for design-session "Context compression exhausted" crashes. The three
compression-retry handlers after an API overflow/413/long-context error (conversation_loop.py
~4229/4488/4747) passed the tool-BLIND messages-only estimate (approx_tokens) to _compress_context,
so hermes-lcm's forced-overflow recovery armed on the message count and missed overflows driven by
tool-schema/system overhead. Now they pass estimate_request_tokens_rough(api_messages, tools=...)
— the same overhead-aware estimator already used at :4580 — so recovery arms on the TRUE request
size; LCM's _overflow_recovery_assembly_cap self-subtracts the overhead so the full request fits.
Empirically validated on real failed session 6dddf1a67b76 (LCM engine, floor=24000/cap=248000):
observed 256,359 >= 248,000 -> arms; recovery 231,313->208,559 msg-tokens -> full request
233,605 < 272,000 FITS. Prior messages-only path did NOT arm (231K < 248K) and crashed.
Durable copy: ~/.hermes/local-patches/optionA-overflow-overhead-aware.patch (survives hermes update
reset). Upstream PR pending. classify_api_error call at :3667 intentionally unchanged (not recovery).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review follow-up: $isStatusRuleOccluded and FloatingOverlays each
enumerated the same six overlay kinds — adding a 7th floating panel
required updating both or the timer gate silently missed it. Extracted
hasFloatingPanel as the shared predicate (completions stays local to
FloatingOverlays; it deliberately never occludes the status rule).
Full ui-tui suite green (1487 tests).
`$isBlocked` answers "is text input suspended", not "is the status rule
covered". appLayout uses it only to hide the input rows (appLayout.tsx:384);
`StatusRulePane` renders outside that guard, at :365 for `at="top"` and :449
for `at="bottom"`. So the previous revision paused FaceTicker /
SessionDuration / IdleSince under prompts that leave the rule fully on
screen — approval, billing, subscription, confirm, clarify, sudo and secret
all render through PromptZone in NORMAL FLOW above ComposerPane
(appOverlays.tsx:58-162, appLayout.tsx:553-568). They push the rule down;
they do not cover it. Freezing a visible clock is a worse bug than the churn
being removed.
Replace it with `$isStatusRuleOccluded`, a narrow derived store over
overlay + ui state covering only what actually paints over the rule:
- `widget` — the modal widget slot renders at viewport level
(ActiveWidgetSlot, sdk/host.tsx:209) so it can anchor the full-screen
absolute `Overlay` against the whole terminal.
- the FloatingOverlays set (modelPicker, pager, petPicker, sessions,
skillsHub, pluginsHub) — but only when `ui.statusBar === 'top'`. That
panel is `position="absolute" bottom="100%"` inside ComposerPane's
relative Box (appOverlays.tsx:387), so it grows UPWARD over the top rule
and can never reach the bottom one.
Deliberately excluded: the PromptZone flow states above; `agents` and
`journey`, which unmount the entire ComposerPane subtree (appLayout.tsx:553)
so React's effect cleanup already clears the intervals; `ambient`, an
in-flow dock; and composer completions, which share the floating grid but
are a render prop that changes per keystroke — re-arming a 1s interval on
every character would restart the countdown each time and starve the tick.
`statusBar: 'off'` needs no branch: StatusRulePane returns null for both
slots, so the timers never mount.
Tests: the store-level cases are re-split into occluding and non-occluding
sets, and an AppLayout-level `describe` mounts the real layout so the rule
sits in its true position — asserting that under approval and sudo the rule
is still rendered AND its clock advances (1m 0s to 1m 30s), that a floating
model picker suppresses the clocks with the rule at the top, and that the
same picker leaves them armed with the rule at the bottom.
`StatusRulePane` renders `StatusRule` outside the `!isBlocked` guard in
appLayout.tsx, so the status rule stays mounted underneath approval,
model-picker, pager, sessions and every other blocking overlay. Its three
timer-driven components keep firing the whole time: `FaceTicker` (glyph,
1s clock, verb rotation), `SessionDuration` (1s) and `IdleSince` (1s).
Every tick re-renders a rule nobody can see, and in an Ink TUI that churn
reads to the user as the dialog flickering.
Gate all three components' interval creation on the existing `$isBlocked`
computed store, so nothing is armed while an overlay covers the rule.
The pause alone would leave the elapsed read-outs frozen at the moment the
overlay opened, so each effect re-seeds `now` from the wall clock when it
re-arms. `SessionDuration` and `IdleSince` already did this; `FaceTicker`
gains the same re-sync. Closing a five-minute overlay now resumes at the
true elapsed time instead of the pre-overlay value.
No new store is introduced — `$isBlocked` already exists and already ORs
the current OverlayState field set.
The PR's skip-live-sessions optimization is partially defeated by
server._shutdown_sessions() registered via atexit (server.py:1172),
which runs on SystemExit after shutdown() returns for the SIGTERM and
stdin_closed paths. The orphan path (os._exit(0)) bypasses atexit.
This is a pre-existing issue — the old finalize-first order had the
same atexit interaction. The comment documents the gap and suggests a
follow-up: gate _shutdown_sessions on not session.get('running').
The drain reserves a slice of the shutdown budget so flush_all_sessions
still runs when in-flight turns outlast the window. But that flush was
unconditional: a session whose turn was still running got its one-shot
_finalize_session spent mid-turn, and the executor.shutdown(wait=False,
cancel_futures=True) immediately after does not join the turn. The
session was then permanently un-finalizable and its active-session lease
had been released out from under live work — the same persistence and
lifecycle race the drain exists to close, just relocated past the
deadline instead of removed.
Give _turn_futures a session association (Future -> sid, the same key
space as server._sessions) at both submit sites, and on deadline expiry
exclude the sids whose futures are still running from the flush. Those
sessions are retained unfinalized and therefore recoverable; sessions
with no live turn finalize exactly as before. The done-callback now pops
under the lock, since a bare dict.pop is not the drop-in set.discard was.
wait semantics, the reserve math and the bounded per-tick sleep are
unchanged, so this adds no shutdown latency. All three shutdown callers
(orphan, sigterm, and the tight stdin_closed wait=2.0 path) funnel
through this one function and are covered.
The drain loop slept a flat 0.05s per tick, so it could overshoot its deadline
by up to one tick and spend part of the reserve withheld for
flush_all_sessions(). For a small `wait` the reserve is itself half the budget,
so a single overshoot can consume all of it: at wait=0.34 the drain budget is
0.17s but the loop requested 4 x 0.05 = 0.20s of sleep.
Clamp each tick to the remaining time. The new test asserts on the summed
*requested* sleep rather than wall-clock, which is deterministic: every sleep is
bounded by the strictly-decreasing remainder, so the total can never exceed the
drain budget regardless of how the scheduler interleaves.
ComputeHost.shutdown() called flush_all_sessions() before its own in-flight
turn drain loop. server._finalize_session latches on session["_finalized"]
and every later call returns immediately, so that one flush was spent while
turns were still producing output: the unflushed tail was never persisted,
commit_memory_session wrote long-term memory from a truncated transcript, the
session's DB row was marked ended while it was live, on_session_end fired with
completed=False/interrupted=True against a running session, and the
active-session lease was released out from under a turn. The drain loop exists
precisely so that mid-turn work survives a teardown; finalizing first defeated
it. Reachable from all three teardown paths: the parent/orphan guard (which
os._exit(0)s immediately after), the SIGTERM/SIGINT handler, and stdin close.
Drain first, then flush. A slice of the caller's budget (_FLUSH_RESERVE_SECS,
never more than half of it so a short explicit wait still gets a real drain) is
withheld from the drain so the flush still runs when turns outlast the window:
HostSupervisor SIGKILLs the host _SHUTDOWN_TIMEOUT_SECS after SIGTERM — 10.0s,
the same value as shutdown()'s default wait — so a drain allowed to consume the
whole budget would leave the durability write racing that kill. `wait` itself is
unchanged, so total shutdown latency and the SIGTERM->SIGKILL margin are
unchanged.
request_restart was calling stop() immediately, so the requesting turn stayed
in the drain wait set and got force-killed at restart_drain_timeout. Wait for
active work to reach zero first, then stop against an idle gateway.
Simplify-pass follow-up on the #69653 salvage: the original code built
these patterns from a name loop; hand-expanding them into 10 literals
lost that single source. Tag-name tuples restore it (adding a 6th
reasoning tag is now a one-place change), and the gnarly named-function
pattern regained a pointer to its step-1c rationale. Byte-equivalence
of every rebuilt pattern verified programmatically (alternation-order
neutrality probed: the \b and > anchors make order irrelevant).
strip_think_blocks passed the same response-scrubbing strings through
re's pattern dispatcher on every response. Skills Guard repeated the
same work for 121 patterns against every scanned line.
Compile the existing expressions once and reuse Pattern.sub/search. Keep
each generic tool-call tag in its own paired expression so mismatched
openers retain their payload while existing stray-closer cleanup remains
unchanged.
Part of #33208
Salvaged from #32713 by @ErnestHysa.
Co-authored-by: ErnestHysa <takis312@hotmail.com>
teknium1 flagged ISSUE_76870_RELATORIO_CAUSA_RAIZ.md as containing
stale metadata (references an unrelated local branch) and asked to
drop the standalone report, keeping only the focused server.py fix
and regression test.
Deferred model switches append a marker and bump history_version at
turn start; the dispatcher was snapshotting history before that
mutation, so the version-mismatch guard rejected the turn's own
result as a stale/concurrent write. Move the snapshot to after
_apply_pending_model_switch/_sync_agent_model_with_config, under
history_lock, so the turn's own preparatory mutation is included in
its baseline while the anti-stale guard still catches real external
writes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A session's YOLO bypass lived only in the in-memory
tools.approval._session_yolo set (or the process-frozen --yolo env
var), so resuming a session in a fresh process silently reverted the
user's /yolo ON — dangerous commands started prompting again.
Persist a yolo_mode flag in the session row's model_config JSON and
restore it on every CLI resume path:
- SessionDB.set_session_yolo() merges the flag into model_config
(same lineage-preserving merge as update_session_runtime_lock);
SessionDB.session_yolo_enabled() reads it back, false on any parse
failure.
- /yolo toggle persists ON and OFF through the new helper; the
compression/branch session-id rotation carries the flag onto the
continuation row.
- --yolo launches record the flag at session creation (agent_init),
and a /yolo toggled before the lazily-created row exists is carried
into the creation-time model_config (_ensure_db_session).
- HermesCLI._restore_session_yolo() re-enables the bypass on startup
--resume/-c, the deferred init path, and mid-chat /resume, with a
visible '⚡ YOLO mode restored from session' notice. No-op under a
frozen process-wide --yolo and never enables on absent/garbage flags.
Copying a Discord thread (or any rich-text selection with images) attached
one or more blank thumbnails and dropped the message text entirely.
Two causes. The clipboard's `text/html` was scraped for inline
`<img src="data:…">` regardless of whether the copy carried its own text —
and what Discord ships beside each image embed is a 32x5 blurhash
placeholder, which is exactly the blank attachment. Then, because any image
blob short-circuited the paste handler, the prose that came with it never
reached the composer.
Inline HTML images now only count for an image-only copy, and are ignored
below a thumbnail-sized floor so spacers and trackers don't attach either. A
mixed paste attaches its real images and still inserts its text.
Also registers pasteAndMatchStyle in the Edit menu — Cmd+Shift+V had no menu
entry, so the chord was never translated into an editor command anywhere in
the app.
When an approval prompt expired without a response, every CLI-side path
collapsed the timeout into the same 'deny' choice as an explicit user
refusal, so the agent was told the user denied the action when the user
simply never answered. The gateway wait already distinguished the two
('timed out without user response... Silence is not consent.'); this
brings the CLI/TUI/ACP surfaces to parity.
- prompt_dangerous_approval(): input()-path expiry now returns a distinct
'timeout' choice (still fail-closed).
- cli.py _approval_callback + hermes_cli/callbacks.py approval_callback:
deadline expiry returns 'timeout' instead of 'deny'.
- check_all_command_guards / _run_approval_gate CLI tails: 'timeout' maps
to outcome='timeout' with a 'timed out without user response... Silence
is not consent.' BLOCKED message (matching the gateway wording);
explicit deny keeps outcome='denied' and gains user_consent=False for
shape parity.
- computer_use: 'timeout' verdict threads through the CLI adapter and
yields a 'prompt timed out — the user did not respond' error instead of
'denied by user'.
- ACP permissions bridge: FutureTimeout returns 'timeout' (other failures
still 'deny'); elicitation maps 'timeout' to 'cancel' like the gateway's
unresolved outcome; codex wire mapping documents deny/timeout→decline.
- write_approval already treats unknown choices as 'stage, not drop', so
a timeout now stages the memory write instead of silently refusing it.
Every timeout path remains fail-closed — the action never runs; only the
classification reported to the agent changes.
Right-click any tab and pick Reload: the pane's content remounts in
place — effects re-run, state resets, measurements are retaken — while
the tab keeps its slot and every other tab is untouched.
A per-pane epoch atom keys the contribution inside the zone body, so
reload never rewrites the layout tree. Both tab menus offer it: the
zone strip menu (tool panels, the file tree, a fresh draft's main tab)
and the session tab menu (tiles + the loaded main tab).
Follow-up for the salvaged #29239: regression test drives the real
gateway _reload_runtime_env_preserving_config_authority() path with a
stale .env TERMINAL_ENV=docker vs config.yaml terminal.backend=local,
and the hermes debug dump test that pinned the old stale-env-wins
symptom now pins the fixed contract (config wins, override line kept
as defense-in-depth for post-load env mutation).
A leftover TERMINAL_ENV in ~/.hermes/.env (written by `hermes setup` or
shell exports) was silently overriding terminal.backend in config.yaml,
so users switching from docker to local saw `hermes config show` agree
with their change while the gateway / cron / batch_runner still ran
against the old backend.
load_hermes_dotenv now re-applies config.yaml's terminal.* values on top
of whatever the .env files set, so the documented source of truth wins
for every entrypoint that goes through the loader.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
Live-run findings from a real fact-checking task (ankylosing spondylitis
genetics, 7 authoritative sources) against the new mode:
- Verbatim check rejected a legitimate quote because web_extract returns
markdown: the MedlinePlus sentence is "including _[ERAP1](https://...)_,
_[IL1A](...)_" on the wire but plain prose to a reader. The agent was
forced onto a weaker evidence fragment — the opposite of the point.
Matching now canonicalizes inline links to their label and drops
emphasis/code markers and backslash escapes on both sides, so quoting
the sentence a reader sees works. Paraphrases are still rejected.
- Escaped asterisks (HLA-B\*27) no longer have to be reproduced in the
quote, so extractor artifacts stop leaking into rendered evidence.
- New `render --replace-in <draft>`: rewrites a draft's Sources block in
place, idempotently. Previously the only path was hand-slicing the
file, which also tripped over the emitted heading being `## Sources`
while the prose said "Sources:".
- verify stats: report the provenance total that the percentage is
actually computed from (cited + [unverified], counted once), and print
the line as `info:` instead of `warn:` when nothing is wrong. The old
line printed 17 cited / 2 unverified next to 72%, which does not
reconcile — a sentence can be both.
- SKILL.md documents the emitted heading, --replace-in, and exactly what
counts as a prose sentence for --min-coverage.
7 new tests (47 total) using the real MedlinePlus/Frontiers markup;
6 sabotage runs, all red.
Extends the citation ledger with evidence-backed fact-checking:
- New `quote` subcommand attaches verbatim supporting quotes to a
source; the quote is rejected unless it appears verbatim
(whitespace/case-insensitive) in the fetched page text, so a
paraphrase or misremembered figure cannot masquerade as evidence.
- `verify --evidence` fails a draft whose cited sources carry no
attached quote.
- `render --style evidence` prints each source's quotes beneath its
URL, showing the claim -> source -> exact-text chain.
- `[unverified]` marker declares model-knowledge claims; counts toward
--min-coverage so provenance is declared for every sentence without
forcing fake citations.
- SKILL.md: new Fact-Checking Mode section + pitfalls; version 1.1.0.
- 10 new tests (40 total), all proven live by sabotage runs.
Covers the fact-checking/evidence-transparency half of #28289.
kshitijk4poor