When Compilers Disagree About UTF‑8

(nemanjatrifunovic.substack.com)

15 points by rbanffy4 days ago

1 comments

  • kstenerud4 days ago
    You could actually use SIMD instructions to detect sequences of bytes with the top bit cleared, thus allowing bulk copies of ASCII text without a per-byte loop.<p>Another optimization would be to take advantage of the fact that codepoint usage tends to cluster around the language of the text. So if you detect usage of 3-byte encodings, chances are you&#x27;ll continue encountering only 3-byte encodings, with the odd ASCII or emoji codepoints. This opens up even more state machine possibilities.
    • duskwuff48 minutes ago
      &gt; So if you detect usage of 3-byte encodings, chances are you&#x27;ll continue encountering only 3-byte encodings, with the odd ASCII or emoji codepoints.<p>Depends on what kind of text you&#x27;re processing. Many languages use the ASCII range for spaces&#x2F;newlines, digits, and punctuation.
      • mort9639 minutes ago
        That, plus markup, be it something XML&#x2F;HTML-like or something Markdown-like.