This is a very good point. Being able to have the Io interface makes testing (ala deterministic simulation testing) much easier. This is one of the big wins I am looking forward to with Io changes is that I can do indepth testing.
1 Like
Thereās also the benefit of hopefully getting a debug Io implementation, to track unclosed fd, network, mutexes etc.
6 Likes
Erm, 106 pointers * 4 bytes may easily chew up 50% of my RAM on a microcontroller.
Leaving that aside, Io is exhibiting similar issues and problems to the Rust executors. The fact that it has āNet, Dir, File, etc.ā abstractions is encasing your system in amber. What happens when I want to extend Io to communication with a memory mapped device or GPU? I need to add more entries and adjust a whole bunch of stuff for the new standards. This is just like I would have to blow apart the executor in Rust and rebuild it from atoms if I need to support external events from a GPU. Part of the reason why I prefer Zig to Rust is precisely thisāI generally donāt have to blow things apart into atoms if my use case is slightly different from what the language developers envisioned.
ānetā and āfileā abstractions are like oil and waterāyou generally need a third ingredient to blend them together properly. Seeing the two of them in the same place is generally a code smell and you probably need to draw the boundaries differently.
8 Likes
Would you consider it a counterpoint that Linux io_uring supports both, fairly uniformly?
2 Likes
io_uring operates on memory buffers and stops there. Net vs File is managed and set up out of band, IIRC.
In addition, TigerBeetle wrote up what they had to do to abstract over it:
And thatās before we start talking about how io_uring is a gigantic security vulnerability magnet.
1 Like
1kB is like half of what even the Atmega 328p has, but I agree with the general point that this sort of mandatory structure seems to contradict the "optimalā in zigās tagline:
Zig is a general-purpose programming language and toolchain for maintaining robust, optimal and reusable software.
But that might be a solvable problem. Maybe Io needs to be split into sub-structures so the basic structure is only like five pointers wide, maybe the structures should be completely separated and passed to functions depending on what they need to do.
That the current implementation is suboptimal does not necessarily invalidate the idea.
Is it? When you want to communicate with a memory mapped device you are implementing something very hardware-specific like a driver, I donāt see why Io would need to be extended in that case, you would just work with the memory directly. It also seems unreasonable to me to try to write an abstraction over direct memory access, that would make it not direct anymore, but I feel thatās what youāre saying as well?
This is 100% fine, the existence of Io does not preclude DMA or doing syscalls directly. All it does make the most idiomatic solution to basic I/O the one that is the most portable and the most testable, which is in line with the āreusableā and ārobustā promise of the tagline.
Or am I missing something critical here?
Iām sorry but I donāt think I understand what you mean by that. Are you saying that in most - if not all - codebases, a function/module should only ever do either networking or file-system access, not both?
6 Likes
Wouldnāt it be nice if the IO features a program needs would be declared in build.zig or something like that?
That would probably allow automatic removal of the unneeded vtable entries and function implementations, resulting in small binaries and at the same time one could see directly which IO features the program uses, thus increasing security?
2 Likes
So if I change my mind during development I would need to modify not only my code but also the build.zig? I donāt like the idea.
2 Likes
Explicit is better than implicit.
I donāt see why changing one line would change productivity, or can explain why you donāt like it?
The main reason for that is that a lot of security work is being done by cloud providers, but since they normally turn iouring off on their end, they often donāt really see the need to invest into iouring.
Thatās also the reason why iouring still isnāt linked up to seccomp.
Hmm I do see some merit in being able to see that a program or library never performs any disk I/O, but this seems quite cumbersome. I think it would be nicer if zigās dead code eliminiation could take care of this in a way where if you donāt use something, you donāt pay the cost for it. The VTable-approach doesnāt allow for that, but this kind of technical discussion is more suited to the original thread:
Also:
Although practicality beats purity.
If I have to specify that a function does I/O by adding a parameter, thatās also explicit.
For the DMA stuff there is some stuff WIP, or at least enough bridge to then be able to use it with std.Io I havenāt tracked it a lot, but as soon as it will go live I will rewrite my firmware with it to see if it helps to make my v4l2 code simpler 
1 Like
Thatās the TODO āmemory-mapped file I/Oā in the tracking issue, right? Or is there something else?
1 Like
If you look at that discussion, Iāve already shared that the current Io was lacking some ways to integrate it nicely with readiness, and control plane like Io, so in the future there might be some better support for that part of the Io
1 Like
1K RAM/8k Flashā16 cents in quantity from TI:
And you donāt have to read Chinese datasheets.
Iāve posted a similar question to zulip but Iām not sure if I was clear enough. Iād be hoping for N~=5 fn pointers with either some kernel-like signature or union(enum) so we could easily switch on those. This would be great not just for testing, but also for VFS and all sorts of interesting server-side only wrappers/decorators. Public view of Zig Software Foundation | Zulip team chat
BTW: Given that Io is a struct, it could still have helper methods so that the consumer API might stay the same, only implementors-side would need to change.
BTW2: I hope that minor perf-drop is not the primary reason for this granularity because Iām not (subjective, I know) using Zig because it has the fastest IO (and IMHO itād close to impossible without going low-level myself), I am using Zig because it allows me to do things I couldnāt do elsewhere.
1 Like
IMHO the current io work is mainly focused figuring out if the abstraction is sound and usable, there are ideas to make it performant.
There is still the proposal for restricted function types. Which was one of the ways to enable stackless code and devirtualize an interface if there is just one implementation. I see no reason why unused function pointers could not become zero size types in this context.
4 Likes
But if thatās the case, then Iād expect it should be able to inline those (hypothetical) switch (op) {} too. But I got it, itās too early to make any opinions, itās just that Iām a bit worried if there will be any round 2.
I think if all implementations are known at compile time (and I would argue that vtables are supposed to be static anyway) you also could see n implementations as equivalent to a single implementation which is a tagged union of n variants.
So while the plans always talked about the āif there is only one implementationā case as special, I think there would be potential to see all implementations as one big batch of code that just needs to continue to work and can be transformed in many non-obvious ways and maybe share a lot of code between implementations.
In general I find the whole discussion a bit premature and I think it would make more sense to wait for 0.16 and then do actual experiments to try and find different ways to optimize it (Ideally through automatic means, instead of manual global refactoring).
I think there are many potential tricks that could be done to optimize things without needing to change the code itself completely, but only how it ends up being optimized and compiled, I think it makes more sense to discuss and explore (mostly) automatic methods (and possibly ways to signal to the compiler that certain things are possible), before considering more manual/refactoring ones.
Basically I think Io is a useful opportunity for trying to answer these questions:
- are there missing features in the language that would allow the compiler to generate more optimal code?
- do the existing backends use all the information they have to generate the best code, or do they miss opportunities?
I think there is potential that some optimizations just arenāt done, because Zig developers havenāt spend enough time yet on coming up with their own optimizations (or are focused on other tasks generally), I think instead we inherited the optimizations from whatever the initial C++ implementation and llvm made easy (I donāt know, but I would imagine)?
The kind of available optimizations influences what code gets written, but at the same time the opposite might be true, if we have more code that benefits from certain optimizations then there is more incentive to write those. So I think it also becomes a question of where do we want to end up in the long term, I think it would be neat if we had great detection and optimization for things that can be ādevirtualizedā.
I put that in quotes because maybe we should start using a different term for it, it seems to me the goal in Zig is more to prevent things from becoming fully virtual in the first place and instead retain more comptime known information.
For example if we have those restricted function pointers, it basically draws a boundary around all the implementation code and makes clear which parts are allowed to be referenced as function pointers from the outside and which parts not. That way everything that isnāt used as a part of some public API surface can be changed internally, allowing to eliminate unused parts.
14 Likes
I think the answer looks more like this than other things which have been suggested in this thread.
I agree and have said so several times now.
Itās understandable that it has its own momentum at this point, but this will be my last contribution to that.
3 Likes