UTF-16 text files (Windows Notepad .txt, PowerShell > redirects) were
refused as binary: the terminal env decodes stdout as UTF-8 with
errors=replace, so their content arrived mangled with U+FFFD and
tripped the binary guard.
ShellFileOperations.read_file now probes the raw bytes via the
backend's Python when the binary guard fires: a BOM or the zero-byte
parity heuristic (derived from VS Code's encoding sniffer, tolerant of
mixed Latin/CJK content) identifies UTF-16 LE/BE, and the file is
transcoded to UTF-8 with CRLF normalized and the BOM stripped. Real
binaries (zeros at both parities), binary extensions, files over
10 MiB, and legacy 8-bit encodings (GBK, Big5) still refuse — a wrong
silent guess is worse than a clear refusal. Works on every shell
backend (local/docker/ssh) since the probe runs via python3 -c.
Tests run against a real LocalEnvironment (E2E, no mocks); sabotage
run confirmed 6/9 fail without the fix.
Teknium