review-fix(gateway): pin split_markdown_atoms empty-fence atoms to main

main's 'and len(current_lines) > 1' fence-close guard is dead (the opening
fence is always already in current_lines), so dropping it is byte-identical
for every input (A/B: 11 empty/nested fence shapes). Add a parity test that
asserts main's atoms for empty ``` blocks so the boundary stays pinned.
This commit is contained in:
Teknium
2026-09-03 09:32:38 -07:00
parent febbe6ae12
commit 58dca1be51
+12
View File
@@ -174,3 +174,15 @@ def test_mixed_emoji_newline_text_utf16_limit():
)
assert all(utf16_len(c) <= 500 for c in chunks)
assert all(c for c in chunks)
# ── split_markdown_atoms: empty ``` ``` blocks (main parity) ─────────────────
def test_empty_fence_block_atoms_match_main():
"""An empty fenced block (``` immediately followed by ```) is its own
atom and the following text starts a fresh one — main's chunk boundaries."""
assert split_markdown_atoms("```\n```\ntext") == ["```\n```", "text"]
assert split_markdown_atoms("```\n```\ncode\n```\ntail") == ["```\n```", "code", "```\ntail"]
assert split_markdown_atoms("```py\n```\n\n```\n```") == ["```py\n```", "```\n```"]
assert split_markdown_atoms("a\n```\ncode\n```\nb") == ["a", "```\ncode\n```", "b"]