Reader delimiters and missing streamUntilDelimiterOrEof

Hey, so today I have been playing with file IO, and while trying to read a file line by line on Windows platform, I’ve noticed that, for a Reader interface, delimiter is only in character long, while for std.mem.tokenize, delimiter is a basically a string.

I’m specifically talking about Windows, because currently it’s not possible to specify “\r\n” as a delimiter. Instead, you basically have to trim the “\r” after using the “\n” as a delimiter, which would be unneccessary if we had a string delimiter for a Reader interface.

Speaking of Reader interface, docs for readUntilDelimiterOrEof this →
“/// Deprecated: use streamUntilDelimiter with ArrayList’s writer instead.”

However, streamUntilDelimiter writes just until it finds the first delimiter, and basically skips rest of the bytes after that first delimiter, so it seems what’s missing is streamUntilDelimiterOrEof.

the delimiter parameter in tokenizer is “any of”, so "\r\n" splits on both \r and \n.

1 Like

Aaah okay, that makes sense actually, thanks for clarifying. I’d be using tokenize completely wrong and probably scratch my head for at least a bit otherwise.

Note that in the latest master versions of Zig, both tokenize and split have been broken apart into 3 versions each (and the unqualified versions have been deprecated). See Split `std.mem.split` and `tokenize` into `sequence`, `any`, and `scalar` versions by squeek502 · Pull Request #15579 · ziglang/zig · GitHub

2 Likes