The Incredible Unicode Mess

[squeek502]

Some relevant links to posts I’ve made:

Why no builtin string type?

Because strings are complicated. That zig-string library you linked, for example, would mishandle comparison: var myString = String.init(allocator); defer myString.deinit(); try myString.concat("Ç"); assert(myString.cmp("Ç")); This assertion would fail, even though the strings appear to be identical. That’s because the first uses Normalization Form D: C (U+0043) + ◌̧ (U+0327), while the second uses Normalization Form C: Ç (U+00C7). To actually compare UTF-8 strings in ways a human might exp…

(also see the other responses in that thread)

Zig’s lack of a string type & invalid values

I don’t think this is the correct way to look at it. Instead, I’d say string literals in Zig are arbitrary sequences of bytes, and Zig makes it convenient to create UTF-8 encoded string literals. For example, take @embedFile: @embedFile(comptime path: []const u8) *const [N:0]u8 This function returns a compile time constant pointer to null-terminated, fixed-size array with length equal to the byte count of the file given by path. The contents of the array are the contents of the file. This i…

Quoting from the first link:

This series of articles by @dude_the_builder details the complication of Unicode well:

• Unicode Basics in Zig - Zig NEWS
• Ziglyph Unicode Wrangling - Zig NEWS
• Unicode String Operations - Zig NEWS

(note that ziglyph has now been superseded by zg)