Review follow-ups on the #25387 salvage:
1. CRITICAL: a cached miss survived out-of-band file creation (terminal
command, external process) for the full 60s TTL — breaking the common
agent pattern 'check for file -> create it -> read it' (live-repro'd).
Serve-side existence guard: one ~free stat before serving a cached
miss; if the path now exists the entry is evicted and the real read
runs. Also fixes the search-root variant (write under a cached-missing
directory). Both mutation-checked.
2. notify_other_tool_call now clears the task's not_found entries too
(belt: the dispatcher calls it for every non-read tool).
3. Tracking parity: the record sites no longer early-return. On upstream,
error results flow through consecutive-loop detection and dedup
bookkeeping; short-circuiting skipped that and broke
TestDedupInvalidationTaskResolution when preceded by
TestSilentFileMisplacementE2E (bisected: the early return at the
read record site was the trigger). Recording is now
side-effect-identical to upstream; serving from the cache remains the
optimization. Also reuse the already-computed _resolved instead of
resolving a second time.
When read_file or search hits a non-existent path, ShellFileOperations
spawns a subprocess to stat the path and another to walk the parent
directory for "did you mean..." suggestions. A typo'd path retried 13
times (observed in the wild) costs 26 subprocess invocations + 13 ls
walks for a result we already know.
Add a per-task negative-result cache keyed by (op, resolved_path) with
a 60s TTL and a hard cap of 500 entries. On hit, return the cached
error JSON immediately and skip the subprocess + suggestion walk.
The cache is namespaced by operation ("read" vs "search") because the
two callers return different error JSON shapes ("File not found:" vs
"Path not found:"). Eviction:
* TTL (60s) — short, so a path that appears later isn't masked.
* write_file / patch on the same path — _invalidate_dedup_for_path
now also drops the negative-cache entry so a freshly-written file
is read from disk on the next call instead of returning a stale
"not found" stub.
Tests in tests/tools/test_file_tools.py cover:
* read cache hit skips the subprocess on retry
* cache is per-task (no cross-task pollution)
* successful reads do not poison the cache
* search cache hit skips the subprocess on retry
* read and search caches are namespaced (different error shapes)
* write_file invalidates the read negative cache
* TTL expiry evicts stale entries
Salvaged from #56982 (@rayjun): the live piece of the PR. The
hermes_cli/config.py get_env_value scope-honoring change and its
test_env_load_cache.py tests were already merged via ed1170cd8b
(#76462) and are dropped here.
tools/xai_http.py::get_env_value wrapped the scope-aware
hermes_cli.config.get_env_value in except Exception + a raw os.environ
fallback — swallowing UnscopedSecretError and borrowing the process
env, so a multiplexed xAI credential read could silently pick up
another profile's XAI_API_KEY. Narrow the except to ImportError (the
only legitimate degraded case) so get_secret's verdict propagates: an
unscoped multiplexed read fails closed, and a scoped miss returns the
default instead of the foreign environ value.
Co-authored-by: rayjun <rayjun0412@gmail.com>
Whole-bug-class follow-up to the profiles/ gate: cron/, platforms/
pairing, and legacy pairing/ ran chown_hermes_tree unconditionally on
every boot with the identical warm-boot cost profile. Same
tree_has_non_hermes_owner gate; find evaluates the top directory first
and -quits on the first mismatch, so a mis-owned tree short-circuits in
O(1) while a clean tree pays one read-only walk instead of a full
chown -R inode rewrite.
The terminal-command guard path loaded config 2-3x per invocation via
load_config(), which pays a defensive deepcopy of the entire config on
every call (~356us of the ~376us warm-cache cost measured on a real
config.yaml). All six swapped call sites were audited read-only — every
caller takes scalar reads or iterates the returned structures; none
mutate (the save path at save_permanent_allowlist keeps load_config) —
so they now use load_config_readonly(), the API built for exactly this
(precedent: #74211, #74322; the one unsafe-site lesson from #56085's
salvage is covered by the mutation audit and a cache-integrity test).
Measured (real config.yaml, warm cache): load_config 376.0us ->
load_config_readonly 19.9us (18.9x); full guard pass
check_all_command_guards('ls -la','local') 930.7us -> 241.8us (3.85x).
Tests: new test_approval_config_readonly.py drives the real functions
against a temp HERMES_HOME — readonly call counts per function, a
no-deepcopy pin for the full guard pass, and cache-identity/integrity
checks. Existing test mocks retargeted from load_config to
load_config_readonly (same injection intent). Note: 6
test_approval_mode_parity failures are pre-existing ordering flakes —
identical with the change stashed on clean main.
kill_started_since duplicated kill_all's collect-under-lock/kill-outside-lock
loop line for line; it is now a thin delegate through new kill_all kwargs
(exclude_ids, source, consume_output). Public signatures unchanged — existing
callers and test monkeypatch seams keep working. kill_process's docstring now
names the deliberate consume_output=True exception for abandoned-turn reaping
so the deviation isn't 'fixed' later.
An agent turn can spawn a long-running background subprocess (e.g.
`next build`) and later be abandoned via inactivity timeout, /stop,
/new, or a client disconnect. Before this fix the gateway interrupted
the agent loop but never touched the subprocess: it kept running
inside the gateway's cgroup, unbounded, until memory pressure starved
the event loop and made every platform/cron look hung (#76115).
The process registry already knew how to kill a process tree — the
missing piece was per-turn ownership: nothing distinguished a process
that predates the turn (must survive), a process the turn started and
finished successfully (must survive), and a process an abandoned turn
left running (must be reaped).
- tools/process_registry.py: snapshot_running_ids() captures a turn's
starting baseline; kill_started_since() reaps only IDs created after
it, scoped to one task_id.
- gateway/turn_context.py: TurnContext carries process_task_id +
process_baseline so the timeout/interrupt paths can reach them.
- gateway/run.py: baseline is snapshotted right before the turn's
executor task starts; the inactivity-timeout path and the explicit
/stop|/new|disconnect interrupt path both reap via the same helper.
A daemon-thread watchdog backs up the asyncio-based timeout poll,
since a starved event loop is exactly the failure mode this bug
causes. The turn's own worker clears its ownership markers the
instant it finishes, closing a race where a /stop landing right
after normal completion could reap a background process the turn
deliberately left running.
Related but insufficient on their own: #37454 (cgroup ExecStopPost
reaper only fires on service restart) and #68915 (orphaned-pipe
grandchild detection, a registry bug not a turn-lifecycle gap).
Neither ties process cleanup to turn abandonment.
- logger.warning when the 6h ceiling stops the heartbeat (matches the
delegate_task stale-stop precedent) so the eventual watchdog reap is
explainable from logs instead of silent
- new mutation-checked test: past the ceiling the heartbeat stops while
the job still completes
- clearer assertion messages (surface res on failure)
- heartbeat loop continues past a raising activity callback instead of
silently stopping (matches delegate_task / touch_activity_if_due
swallow-and-continue semantics) — one transient error must not drop
watchdog protection for the rest of a long job
- hard 6h elapsed ceiling so a wedged job under HERMES_CRON_TIMEOUT=0
(unlimited child watchdog) cannot mask the gateway watchdog forever
- public get_activity_callback() accessor in tools/environments/base.py
instead of importing the private _get_activity_callback cross-module
- tests: deterministic heartbeat test (event-gated, no timing sleep),
no-callback test now asserts the thread is truly never created, new
exception-survival guard; dead started event removed
- fix comment: delegate_task heartbeat cadence is 30s, not 10s
Follow-up to the salvaged #76000 guard:
- refuse installing a skill INTO an existing skill directory (hybrid
skill-plus-category dirs whose later update/uninstall rmtree would
destroy the nested skill — sibling case of #75983)
- refuse a stray regular file at the install path with the caller's
ValueError contract instead of an uncaught NotADirectoryError
- regression tests: nested-only category (skills at depth >= 2),
category-inside-skill, file collision
Start from main's 13 tests (renamed test_openai_available_reflects_key
to test_openai_available_reflects_audio_key_resolution, added 4 new
tests for xai oauth, elevenlabs secret resolver, openai configured
key, stream cap). Append 12 new regression tests from PR #71084 for
the prefetch pipeline, PCM misalignment, and PortAudio resilience.
Patch platform.system in stream-path tests for main's macOS guard.
Three lower-severity core-tool robustness fixes from a targeted audit, each
reproduced live:
1. terminal_tool did not validate non-positive timeouts. 'timeout or default'
silently coerced 0 to the config default (0 can't mean 'no timeout'), and a
negative value is truthy so it flowed into 'deadline = now + timeout' and
fired an immediate '-Ns' timeout. Reject timeout <= 0 with a clear message.
2. fuzzy_find_and_replace accepted a whitespace-only old_string, which matches
trivially (blank line / run of spaces) and mass-replaces under replace_all
or raises an opaque ambiguity error. Reject it alongside the empty check.
3. The '/private/var/' sensitive-path prefix over-blocked ALL macOS temp-file
writes: , /tmp, and /var/folders realpath into /private/var/folders
on macOS (and paths are resolved through symlinks), and /private/var/tmp is
a normal temp dir. Narrowed to the genuinely-sensitive subtrees
(/private/var/db, /private/var/root); /etc and /private/etc stay blocked.
All verified with sabotage-checked regression tests. 85 terminal/fuzzy/file
tests pass; normal timeouts, legit replacements, and /var + /boot + /etc
blocking are unaffected.
Two V4A parse/validate bugs found in a core-tools audit, reproduced live:
1. CRLF patch body injected stray carriage returns. parse_v4a_patch split
on '\n' only, so a CRLF-encoded patch kept '\r' inside every HunkLine
and wrote mixed line endings into an LF file; the anchored Begin/End
markers could also fail to match because of the trailing '\r'. Strip a
trailing '\r' from each line at split time.
2. Move-then-Update of the same file was rejected. _validate_operations read
the UPDATE target from disk before the MOVE ran, so 'Move a->b' + 'Update
b' failed validation with 'b: file not found'. Added a small pending-move
overlay so UPDATE/DELETE/MOVE reads during validation see prior ops'
effects (moved-in destinations resolve, moved-away sources read as gone),
while a genuine 'destination already exists' conflict is still caught.
Both verified with sabotage-checked regression tests. 113 patch/fuzzy/file
tests pass.
Strategy 9 (context_aware, the last-resort fuzzy strategy used by
patch_replace, V4A UPDATE hunks, and skill_manage) had two serious flaws,
both reproduced live against current main:
1. CORRECTNESS: it accepted a block when >=50% of its lines were >=0.80
similar. A 2-line pattern with one real line and one garbage line matched,
silently deleting the non-matching line and persisting a wrong edit as
success. Now requires the first AND last lines to anchor-match and EVERY
non-blank pattern line to be >=0.80 similar — one garbage line disqualifies
the block.
2. PERFORMANCE: it scored every content window with per-line SequenceMatcher,
so every failed match paid O(file_lines x pattern_lines) — measured ~5.5s
for a single 40-line no-match on a 10k-line file, per hunk. The first/last
line anchor pre-filter skips non-candidate windows: same case now ~160ms
(34x faster).
Also gate replace_all: a similarity-based strategy (block_anchor,
context_aware) with multiple matches under replace_all would overwrite every
approximate block, not just exact ones. Now refused with a clear error
directing the caller to precise text.
All verified with sabotage-checked regression tests (fail against the old
50% logic). 158 file/patch/fuzzy tests pass; legit fuzzy edits (indent drift,
unique near-match) unaffected.
Two DATA-LOSS bugs in ShellFileOperations found in a core-tools audit,
each reproduced live against current main:
1. Non-UTF-8 file content silently corrupted on read->write. The terminal
env decodes stdout with errors='replace', so a latin-1/8859 file's bytes
arrive as U+FFFD before _is_likely_binary inspects them. U+FFFD is
'printable', so the >30%-non-printable check never flagged it, and the
agent would read the mojibake and write it back, permanently replacing the
original bytes. Fix: treat a sample containing U+FFFD as binary (read-only).
2. Writing through a symlink destroyed the link and orphaned the target. The
atomic temp-file + 'mv -f' swap replaced the symlink itself with a plain
file; the real target was never updated. Fix: resolve the link with
readlink -f/realpath first and recompute the temp dir from the resolved
target so the mv stays same-filesystem atomic. Broken links fall back to
the original path (no regression).
Both verified with sabotage-checked regression tests (fail without the fix).
Proper UTF-8 text (incl. non-ASCII) and plain-file writes are unaffected.
Two independent HIGH-severity correctness bugs found in a core-tools audit,
each reproduced live against current main:
1. Read-dedup was never evicted after a write on non-default tasks.
_invalidate_dedup_for_path looked up the read-tracker under the correct
task_id but resolved the path with _resolve_path(filepath) — which
DEFAULTS task_id='default'. The dedup cache is keyed by the task-resolved
absolute path, so for any task whose workspace cwd differs from the process
cwd (every -w worktree / Desktop / ACP session using relative paths) the
computed key never matched and the stale entry was never removed. A
read_file after a write_file/patch could then return the OLD content stub
when mtime coincided. Fix: pass task_id through.
2. A per-command workdir override permanently hijacked the session cwd.
The post-command dual-write unconditionally recorded env.cwd (stamped to
the transient workdir) into the durable session-cwd store, so every later
command that omitted workdir inherited the one-off directory — contradicting
the documented 'Working directory for this command' contract. Fix: skip the
session-cwd record when workdir was explicitly supplied.
Both verified with sabotage-checked regression tests (fail without the fix).
patch_tool resolved V4A header paths against the task workspace for
locking, staleness, and reporting, but handed the original (often
relative) patch text to file_ops.patch_v4a — which re-resolved headers
against the backend env's own cwd. When the two diverge (the git-worktree
cwd bug), a relative header landed in a different directory than
everything the tool locked and reported: a silent wrong-file write.
Rewrite Update/Add/Delete/Move File headers to the resolved absolute
paths before apply, only for host-filesystem backends (container/remote
namespaces keep their own paths). Header patterns mirror patch_parser
(no-space ***Update File: form) and cover Move File: src -> dst.
Salvage of #53176 by @necoweb3, reimplemented onto current main (the
original branch predates the sensitive-path/Move-header extraction and
per-path locking now in patch_tool).
Co-authored-by: necoweb3 <sswdarius@gmail.com>
The boundary scan in parse_v4a_patch used substring matching, so a
content line mentioning "*** End Patch" (docs about the patch format,
nested patch text) truncated the patch, and "*** Begin Patch" in
content reset the start boundary — silently dropping already-parsed
operations while reporting success. Match only whole-line markers at
column 0, preserving the no-space "***Begin Patch" tolerance.
Replace the bare /init ENTRYPOINT with entrypoint-dispatch.sh: exec
/init + main-wrapper when the image owns PID 1, fall back to a direct
stage2 bootstrap (with the s6 helper PATH restored) on wrapped runtimes
where s6-overlay-suexec would abort with 'can only run as pid 1'
(Fly Machines, docker run --init, podman/FreeBSD setups).
Cherry-picked from PR #43763 by @konsisumer, conflicts with current
main resolved (tests/test_dockerfile_tini_compat_shim.py was moved to
tests/docker/, container_boot argv tests were reshaped upstream).
Fixes#38349
Two amplifiers behind the 779K api.osv.dev DNS queries/16h report:
1. tools/osv_check.py: check_package_for_malware() hit OSV on EVERY
call. MCP reconnect ladders, stdio recycles, and parked-server
self-probes re-run the preflight for the same package on every spawn
attempt, so a flapping server became a sustained OSV query/DNS
stream. Verdicts (clean or blocked) are now cached for 1h
(OSV_CHECK_CACHE_TTL to tune); network failures stay uncached so
fail-open never masks a real advisory once connectivity returns.
2. hermes_cli/security_audit.py: cmd_security_audit() ran full
component discovery twice per audit (_count_components + run_audit).
Discovery now runs once via _discover_components() and run_audit()
accepts the pre-discovered list.
Both regression tests fail against the previous code (verified via
sabotage run).
Let terminal keys explicitly present in config.yaml override matching stale TERMINAL_* values while preserving environment values for omitted keys. Merged defaults remain backfill-only.
Exercise the real config.yaml to _get_env_config path for backend selection, partial terminal sections, matching-key overrides, environment fallback, one-shot bridging, and config read failures.
Closes#71137
Google API and authentication packages permit vulnerable httplib2 and pyasn1
transitives, while the Workspace and Google Chat runtime installers previously
treated any importable version as sufficient. Existing environments could
therefore remain vulnerable after the project dependency pins were repaired.
Carry the fixed versions through the Google and Vertex extras, lazy feature
requirements, lockfile, and both runtime installers. Route the documented
Google Chat installation path through its maintained secure requirements
instead of an unconstrained direct pip command.
Detect stale distributions, install only unsatisfied requirements, and verify
the result before continuing. Behavioral tests cover those repair invariants
without freezing manifests, lockfiles, or complete package sets.
Related #72108
Extracted from #72840
Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
Three fixes for concurrent MCP server spawn races in register_mcp_servers()
and discover_mcp_tools():
1. register_mcp_servers: add k not in _server_connecting guard to the
new_servers filter. Without this, a concurrent second call sees the
same servers as 'new' and spawns duplicate stdio subprocesses.
2. discover_mcp_tools: same _server_connecting guard in the
new_server_names filter. This entry point is called from CLI, TUI,
gateway, and cron — any two racing would double-spawn.
3. Stale _server_connecting cleanup on TimeoutError/InterruptedError.
When _run_on_mcp_loop times out or is interrupted, _discover_all's
gather may not have finished, leaving entries stranded in
_server_connecting that block future reconnection attempts. The
cleanup clears only entries added by this call (not external ones),
logs a warning, and records connect errors.
Salvage of #58879 by @nanami7777777 (superset of #58867 by @liuhao1024).
Adapted to current main which has evolved significantly since July 5.
Closes#58862Closes#58867Closes#58879
Two fixes for the Skills Hub "View source" links on ClawHub skills:
1. Source URL generation was missing the required {owner} segment —
https://clawhub.ai/skills/{slug} → 404. Correct format is
https://clawhub.ai/{owner}/skills/{slug}. When the owner handle is
unavailable, source_url is now "" (card omits the button) instead of
emitting a broken link.
2. _fetch_owner_handle() previously delegated to _get_json() which
returned None on any non-200 response with no retry. Under HTTP 429
rate-limiting the "50 consecutive failures" safety rail in
enrich_owners() fired immediately — the documented claim "Respects
HTTP 429 rate-limit responses with exponential backoff" was not
actually implemented. Now has its own retry loop: 3 attempts, honours
Retry-After on 429, exponential backoff on 5xx/transport errors, no
retry on 4xx.
Changes:
- tools/skills_hub.py: _coerce_skill_payload carries owner from top-level
response; inspect() captures owner from detail API; _fetch_owner_handle()
added with bounded retry/backoff; enrich_owners() batch method with
safety rails (30 workers, early termination at 50 consecutive failures).
- website/scripts/extract-skills.py: _source_url() reads extra["owner"]
for ClawHub.
- scripts/build_skills_index.py: batch enrichment step after crawling.
- tests: 35 URL/enrichment tests + 7 retry tests (42 total).
Signed-off-by: dongjiang <dongjiang1989@126.com>
Two holes in _strip_cron_safe_constructs (one a regression from
70411a615, two days old):
1. The [^\n]* tail erased everything after api.github.com on the line,
so a payload smuggled after ; && or | was never scanned. A cron
prompt carrying a benign-looking GitHub curl followed by
'cat ~/.hermes/.env' or 'rm -rf /' passed the scanner and persisted
(verified end-to-end through the cronjob tool). Bound the tail to
the URL path ([^\s;&|]*), so same-line payloads survive the strip.
2. The (?:/|\b) host boundary treated lookalike authorities
(api.github.com.evil.com, api.github.com@evil.com) as the trusted
GitHub construct, erasing even exfil of the GitHub token itself to a
non-GitHub host. Require the exact host followed by /, whitespace,
or end.
Also add SSH private-key files to the read_secrets pattern — a
coverage gap found during adversarial testing (cat ~/.ssh/id_rsa was
invisible to the scanner even outside the exemption).
Port from nanocoai/nanoclaw#2748: Docker's built-in 64 MB /dev/shm silently
breaks shared-memory-hungry workloads inside the sandbox — Chromium/Playwright
renderers crash tabs, and PyTorch DataLoader workers die with 'bus error' /
'insufficient shared memory'. tmpfs is lazily allocated, so the higher ceiling
costs nothing until actually used, and usage still counts against the
container's --memory cgroup limit.
- tools/environments/docker.py: --shm-size 1g in resource args (not
cgroup-gated; tmpfs mount option). Skipped when docker_extra_args already
sets --shm-size, or when configured empty/'0' (Docker default).
- terminal.docker_shm_size config key (DEFAULT_CONFIG + all three
config->TERMINAL_DOCKER_SHM_SIZE env bridges: CLI, gateway, config.py map)
- tests: default emit, custom value, opt-out, extra_args precedence,
helper edge cases (sabotage-verified: default/custom tests fail without
the emit)
Port from openclaw/openclaw#112954. The redactor knew GitHub, Slack,
Google, Stripe, AWS access-key-ID and ~25 other vendor prefixes but had
zero GitLab coverage — glpat-/gloas-/gldt-/glrt-/glrtr-/glcbt-/glptt-/
glft-/glimt-/glagent-/glsoat-/glffct-/glwt- tokens and legacy GR1348941
runner registration tokens passed through display and log surfaces
verbatim. Follow-up explicitly invited when #4541 was closed.
Each pattern keeps a full literal prefix so the _PREFIX_SUBSTRINGS
pre-screen (derived at module load) stays false-negative-free; routable
runner tokens allow dotted segments. Sibling site: skills_guard's
credential-exposure scan gains a gitlab_token_leaked pattern.
Port from openclaw/openclaw#112325: multibyte UTF-8 characters split
across a 4096-byte pipe or PTY read boundary were decoded statelessly
per chunk with errors='replace', corrupting both halves into U+FFFD
mojibake in background process output (poll/log/wait/completion
notifications). The foreground path already used an incremental decoder
(tools/environments/base.py::_wait_for_process); this applies the same
treatment to the background reader loops:
- _reader_loop (select and blocking paths): one
codecs.getincrementaldecoder('utf-8') per reader holds partial
sequences across chunks; the finally block flushes a truncated tail
as a single U+FFFD instead of dropping it.
- _pty_reader_loop: same treatment for ptyprocess byte chunks
(pywinpty str chunks pass through unchanged).
Genuinely invalid bytes keep errors='replace' behavior.
Inspired by Claude Code v2.1.219: MCP config string values with hidden
leading/trailing whitespace (pasted tokens with trailing newlines, URLs
with leading spaces) now trigger a startup warning naming the server and
the dotted key path, instead of failing later as an opaque auth/connect
error.
Advisory only: values are never mutated, secrets are never logged (only
key paths), and warnings dedupe to once per process per (server, path).
Checked after ${VAR} interpolation so whitespace inside referenced env
vars is caught too.
* fix(vision): mount images/ upload dir into sandboxes and permit host read (#69575)
Desktop, clipboard, and PDF uploads land in the flat top-level
HERMES_HOME/images/ dir, but Docker sandboxes only mounted the cache/
subtree and the vision resolver only permitted host reads from the media
caches. So vision_analyze on any desktop-app upload failed under a Docker
backend with "not reachable inside the sandbox".
- Add ("images", "images") to _CACHE_DIRS so the uploads dir is bind-mounted
into sandbox containers through the existing profile-scoped cache-mount and
reverse-mapping mechanism.
- Add home/"images" to _media_cache_roots() so the non-local host-read
allowlist permits reading uploads directly from the host filesystem.
- Cover the mount entry, the container path mapping, and the Docker-mode
resolver read for a profile-scoped upload.
Co-authored-by: JonthanaHanh <92574114+JonthanaHanh@users.noreply.github.com>
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
* fix(tui_gateway): write image uploads under the session's profile home (#69575)
The attach RPCs (image.attach_bytes, clipboard.paste, pdf.attach) wrote
uploads to the gateway's module-cached launch home via _hermes_home/"images".
Those RPCs run before prompt.submit installs the session's profile HERMES_HOME
override, so in a multi-profile / root-gateway deployment the file landed in
the launch home while the sandbox mount and the vision host-read allowlist
both resolve the session profile's images/ at run time — the agent could
never see the upload it was handed.
Add _session_images_dir(session), which anchors the write on the session's
stored profile_home when present (matching the mount/read scope) and falls
back to the launch home otherwise. Route both write sites through it, keeping
per-profile isolation.
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
---------
Co-authored-by: JonthanaHanh <92574114+JonthanaHanh@users.noreply.github.com>
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Follow-ups on top of #70888's cherry-picked fix:
- Replace the $((0666 & ~0$u)) shell arithmetic with POSIX who-less
'chmod "=rw"'. zsh (reachable via _find_bash's $SHELL fallback on
bash-less hosts) parses leading-zero constants as decimal and silently
chmods a garbage mode (e.g. 0210); the symbolic form is spec-identical
across bash/dash/busybox-ash/zsh and degrades to mktemp's 0600
(pre-fix behavior) rather than corrupting perms if chmod rejects it.
- Move the new-file chmod after the content stream so the temp file
stays owner-writable while cat runs.
- Run the chmod on a '[ ! -e "$t" ]' check after cat instead of the
stat/else branch, keeping the overwrite path untouched.
- Update the stale perms comment #70856 called out (new files did NOT
land with default umask perms pre-fix).
- Tests: select the atomic-write script by content instead of call
order (the previous last-call capture only worked because the bare
MagicMock's falsy-exit early return suppressed later execs), assert
behavior at explicit umasks 0022/0002/0077 via parametrize, add an
overwrite mode-preservation regression guard, and dedupe the
real-subprocess env fake into make_real_subprocess_env() shared with
TestSearchFilesFallbackHiddenPaths.
(webtecnica's email mapping already exists in contributors/emails/ on
current main; the PR's check-attribution red was stale-base only.)
# Conflicts:
# tests/tools/test_file_operations.py
Org-shared skills were unusable past the first propose. Three defects, one
root cause plus two that it masked.
ROOT CAUSE — org reads went to the personal endpoint.
`SyncClient.get_refs()` / `get_object()` only ever called `/v1/sync/refs`
and `/v1/sync/objects/:hash`. Those routes are hard-scoped server-side to
the token's own owner, so asking them for `refs/org/<id>/` returns the
caller's PERSONAL refs rather than an error, and org objects 404. Both org
call sites read org state through them:
- `pull_org_skills` resolved head=None for a populated org and reported
`{"ok": true, "head": null, "updated": []}` — org skills silently never
arrived, which reads as "my org has no skills" rather than as a failure.
- `propose_skill` resolved base_head=None, so the FIRST propose to an org
succeeded by accident (`from: null` happened to be correct) and EVERY
later one CAS'd against a head it had never seen -> 409 -> a raw
`SyncConflict` traceback. Worse, it built its root from an empty skill
map, so a landed CAS would have REPLACED the org set rather than splicing
into it — the 409 was accidentally preventing data loss.
Fix: `org_scope=True` on `get_refs`/`get_object`, threaded through
`get_commit_json`, `get_tree_json`, `_root_tree_of_commit`,
`_skill_trees_of_root`, and `materialize_tree` — walking an org commit needs
the org route on every hop, not just the first. Both org call sites now go
through one `_read_org_head()` helper.
ALSO FIXED
- `propose_skill` retries on conflict. When the org HEAD moves between the
read and the CAS (another member proposing, an admin merging), it
re-splices this one skill onto the NEW head and retries, bounded at 5
attempts. Re-splicing rather than replaying the old root is what stops a
concurrent proposal being dropped.
- An empty `actual` in a 409 means "the ref does not exist", not "here is a
commit". `SyncConflict` normalizes "" to None in its constructor, and the
personal push path redoes the CAS as a create instead of fetching "" as an
object — which surfaced as the baffling `object not found` (doubled
space). This is what a client hits after switching sync planes, since
`.sync_state` is not environment-scoped and carries a foreign head.
THE MOCK WAS THE REASON THIS SHIPPED
The test mock served org refs and org objects off the personal routes, so
21 org tests passed against a client that could not work against the real
plane. The mock now mirrors production: `/v1/sync/org/refs` and
`/v1/sync/org/objects/:hash` exist, org objects live in a separate scope,
and the personal routes refuse org content. Two existing tests had to be
corrected to assert against the org scope — they had been passing on the
mock's over-permissiveness.
Tests: 5 new (org head invisible on the personal route; second propose
splices and preserves the first; pull resolves a real org head; empty
`actual` -> None; push recovers from a stale cross-plane head). Verified
they FAIL without the fix: reverting just `_read_org_head` to the personal
route fails the second-propose test and the pre-existing splice test.
1278 passed / 0 failed across 54 suites via scripts/run_tests.sh.
Verified against PRODUCTION with a real org token, not just the mock:
- `pull_org_skills` -> head `sha256:1adf9333…`, materialized
`software-development/gateway-gateway-connector` into the `_org` mirror
(was head=None, updated=[]).
- A second `hermes sync propose` succeeded where it previously raised, and
the org set afterwards contains BOTH skills with the new commit
descending from the first.
send_video got the 60s read_timeout but send_voice/send_audio/send_photo/
send_document/send_media_group/send_animation upload through the same PTB
request path and hit the same server-side processing wait before the
response arrives. Same class, all sites: they all pass
_MEDIA_SEND_READ_TIMEOUT now. Also drops an unused test helper.
kshitij