I'm too dumb for Zig's new IO interface

Media
21

file is redundant, the File.Reader has it already.

1 Like
22

I noticed that the handy delimiter function everybody wants is missing (the one you would use in a quick script to process a file line-by-line). Will remedy this, but I also need to address the problem that the other delimiter functions need to decouple the maximum line size from the buffer capacity.

Also noticed the situation where itā€™s surprising that streamDelimiter does not discard the delimiter from the input.

Letā€™s keep looking at these APIs with a critical eye for improvement.

21 Likes
23

It must be sometimes difficult to see al the issues, questions, critique.
So as compensation i would like to add: I absolutely love Zig. This is the language / compiler I craved for since I started programming looooooooong time ago.

8 Likes
24

If we are looking at the std.io.Reader API, i find it really common that i end reading input on a sequence, not a specific single delimiter.
(for example more complex utf-8 character, protocol byte sequence or the common \r\n)

Of course i just implement this kind of reading on my own, but i cant deny that it is a bit tricky, because when the buffered input ends mid the specified sequence, there is no single best solution. (or i do not see a clear winner)
Would be nice to have such common use-case covered inside the interface.

Do you think it would be possible, or is it just too specific to a task?

Robert :blush:

25

Thank you for the kind words.

The biggest challenge for me is that criticism often falls into one of two categories:

  1. People who use Zig, observe problems, and take the time to report their experience, whether on the issue tracker, or in a community space like this.
  2. People who see programming languages as some kind of competition and are pretending to offer constructive criticism but actually are acting in bad faith, trying to damage the project so that their preferred horse in the race ā€œwinsā€.

After enough interactions with people in category (2), there is a strong danger of becoming jaded. My 2025 New Yearā€™s resolution was ā€œbe less cynical and more optimisticā€.

Whatā€™s nice about a forum like this is that you get to know people. That means I can let my guard down when Iā€™m interacting with you or other people who Iā€™ve seen many times over the course of, letā€™s seeā€¦ three years now!

36 Likes
26

Hello Andrew, and thank you for creating Zig, I am using it every day and so far it has been both fun & useful :slight_smile:

Have you considered putting some of that helper logic into the new IO Interface itself? Given that IO interface is the new thing and it will happen in next release, why not make it also a primary gateway for all sorts of every-day things and operations?

Because if you need to pass &io everywhere you could also just call methods on Ioā€¦

I had this idea during my kqueue deep-dive this winter but I thought it would be a no-go for you because it would require such methods to accept in/out: anytype but you also seem to have io.async(fun: anytype, args: anytype) ad io.select(s: anytype) accepting any type already, so maybe itā€™s something to consider? If you havenā€™t already.

27

what helper logic, you are replying to a side comment about criticism, I donā€™t think the IO interface can help you with that :p.

If youā€™re talking about reading till delimiters, that makes no sense to be in Io, keep the stream reading logic in the stream reading interface.

ah but you do, for generic io stuff, you pass io to other things because they need it to do their thing, why is their thing not in Io, because compartmentalising distinct features is usefull for comprehension, maintanance and improvments, also because while it might do io/async operations its doing something else on top of that.
reader/writer are a good example because their streams can be to/from fixed/dynamic memory not just io. io would be required for implementations that require io operations, but thats only a subset of streams, even if it is a large subset.

28

I guess I am trying to say several things at once (and any of them might be interesting by itself):

  1. When Io is merged and it needs to be passed to every read/write method call, then what is really the Reader/Writer purpose besides holding the vtable and its buffer? All the methods which are implemented there, could be also implemented in the Io interface, because Reader and Writer will be useless without Io.

  2. anytype means that Io.read/write() could have special treatment for deprecated/special streams. I am not experienced enough in that area but I think some OS-level things like pipes/timers could benefit from this? You could also detect T.reader/writer(buf) methods and call them automatically (fs.File, ArrayList, ā€¦)

  3. There is (IMHO) a space for various one-shot utilities/common operations and I donā€™t have a (good) specific example but I think it makes sense to structure the API in terms of lots of various operation which then work over few data-types.

  4. One interesting example could be something like print-read part of the REPL loop, where you know you can reuse the same buffer, you know when to flush, and all you need is just out: anytype, in: anytype and voila, it can be either existing Reader, or something convertible, and in that case, they would be stack-owned and sharing the same buffer. Not sure if this particular thing should be in std, but I hope it illustrates my point enough.

  5. I like the direction, but I think something important is still missing - I canā€™t point the finger but I am in the process of porting to 0.15.1 and I am also having a hard time.

1 Like
29

Code that deals with streams (e.g. parsers, like std.json) wonā€™t need a reference to Io, they will just use Reader/Writer (which might or might not be wired to actual I/O).

Case in point from my sentence above, and as a counter example: std.Io.Writer.Allocating (which buffers writes in heap-allocated memory).

1 Like
30

Ok, I have obviously misunderstood something, feel free to ignore my previous post :slight_smile:

31

I literally created his account to leave this comment lol

The thing that I cannot get my head around is why you need to access the interface directly. Iā€™m sure this is just Java brain, but when I hear the word ā€œinterfaceā€, my expectation is that the file writer implements the interface. So in my mind I should just be able to create the writer and pass it to anything that accepts said interface.

Thereā€™s something fundamental with the way Zig handles this that Iā€™m clearly missingā€¦

1 Like
32

Welcome to ā€œpost-writergateā€ and this forum.

2 Likes
33

Conceptually, File does indeed implement the Writer interface, here (and the enclosing Writer struct)

You mention Java, where you would do something like class File implements Writer. Since Zig lacks interfaces as a language level construct, you manage everything yourself using one of a few available techniques (function table + type erasure / @fieldParentPointer, etc), whereas Java and some other languages will generate this stuff for you. There are pros and cons for both, but Zig is a low-level language that allows you to pick the most efficient approach for the situation at hand. Like for the new Writer interface, the buffer lives in the interface which would be hard to do in some languages having interfaces as a first-class construct.

3 Likes
34

for what it is worth, it confused the hell out of me, too. however, I feel I am coming out that mind shift a better programmer. once it clicksā€¦ :exploding_head:

3 Likes
35

The way java (and many languages) implement dynamic polymorphism is very similar to how itā€™s done in Zig, except that higher level languages hide the details behind language constructs whereas Zig allows you to implement them directly from lower level primitives.

36

Youā€™ve gotten good answers. I just have a little more to add about the comparison to Java specifically, as I assume youā€™re using it.

An interface in Java can be implemented internally, in the JVM, as an object with a vtable, similarly to the Zig interfaces being discussed. However, Java also has a git compiler and when possible (e.g., there is only one implementor of the interface being used) it can remove the vtable so the implementorā€™s functions are called directly. This removes the indirection and is a pretty nice feature that Java deserves credit for.

Zig OTOH has comptime generics where there is no vtable and code is monomorphized (duplicated and optimized for the concrete types and how theyā€™re used). With Zig, you choose which approach to use, based on your need for these specific types of optimizations, vs the simplicity of a non-generic API.

Interestingly, the Zig IO interfaces have both high performance and the simplicity of a non-generic API. The indirection of the vtable is not a performance issue because of buffering in these interfaces. The vtable functions are only called when needing to fill the buffer to read, or flush the buffer to write. When that happens, a system call to do the IO will occur, and the cost of that will dwarf the cost of the vtable indirection. Therefore, the vtable interface is perfect for this use case. Thatā€™s the kind of tradeoff you can make when choosing whether to use generics or interfaces in your own Zig abstractions.

So the difference is whether you can choose the implementation and get exactly what you asked for and what you need (with Zig). Or whether you leave it to the JVM and git compiler to optimize in some case but not others, without knowing whether youā€™ll get the optimizations you need. In many cases you donā€™t care, but in other cases you really do want those specific optimizations.

6 Likes
37

Okay, so potential stupidity ahead, but Iā€™m gonna do my bestā€¦

I think I have been getting some terms mixed up. I keep thinking of std.Io.Writer as the interface, but Iā€™m realizing now that might be wrong.

Is it correct to say the ā€œinterfaceā€ is represented by the vtable, which has drain, sendToFile, flush, and rebase? So File implements those functions, passes the implementations to std.Io.Writer, which then uses them in itā€™s own methods to execute the actual I/O operations?

38

std.fs.File.Writer has a field interface. The vtable and the buffer is in this field. You use a pointer to the interface to do the writing. It writes to the buffer and then calls back to std.fs.File.Writer through vtable. You make sure you use the pointer to the interface field and not a copy.

1 Like
39

I get what youā€™re saying about Java, and I get the advantages in Zig implementation, but I guess I"m just not familiar enough with low level implementations of V tables.

For example, std.Io.Writer has a ton of stuff on it, so when thatā€™s called an ā€œinterfaceā€, I expect something somewhere else to actually be implementing those. But from the source code it looks like std.io.writer is the implementer, and itā€™s just getting certain things (like drain) passed to it from the vtable coming from File (in this specific example).

40

Take a look at my longer post explaining this: Zig 0.15.1 reader/writer: Don't make copies of @fieldParentPtr()-based interfaces

welcome to Ziggit!

1 Like