Reader return EndOfStream before remaining bytes

Explain
21

I see! Reader.zig:680 has a branch so if it copied nothing from content (because interface buffer len is 0), it just defaults to readVec and uses the given buffer.

True, those two was me trusting ZLS a bit too much. I’ll stop consenting to what it tells me to do in the future.

Never!

That was an interesting read. As a game dev I have never had to think about these things. I also read mem.zig:3348:isDelimiter() function, which indeed just cares about T, and Writer.zig:585 which to my surprise supports waaaaaay more types of output that I was expecting.

test {
    const buff = "ohÃno";
    var it = std.mem.tokenizeAny(u8, buff, "ñ");
    while (it.next()) |s|
        log("[{s}]", .{s});
    // [oh][�no]
}

Cool to see the second byte of à as � since it does not coincide with the second byte of ñ! Changing à for ± changes the side � appears :smiley:

Mainly because I didn’t write it for a code interview. In any other situation I’d either write it separately or kill whoever wrote it.

22

Re: " the second byte of à as � since it does not coincide with the second byte of ñ !"

Oh but this is Unicode, and if using combining ~, it is the same. One needs to consider normalization forms.

1 Like
23

You do seem to mix up different Unicode concepts. I was not talking about combining anything.

test {
    std.debug.print("combining tilde after n: {s}\n", .{ [3]u8{ 'n', 0xcc, 0x83 } });
    std.debug.print("spanish letter as a single unicode character: {s}\n", .{ [2]u8{ 0xc3, 0xb1 } });
    std.debug.print("invalid utf-8: {s}\n", .{ [1]u8{ 0xb1 } });
}

Output:

combining tilde after n: ñ
spanish letter as a single unicode character: ñ
invalid utf-8: �

The � has nothing to do with Unicode itself and everything to do with the way utf-8 encodes characters as multi-byte sequences. Normalization would not make ñ a single-byte character.

1 Like