Async, stacks and colors revisited

This is likely to go essay style, so if you are looking for a fast read, this is not it.

First this: I love std.Io and it’s concurrency/async concept. It’s beautiful. It’s unfinished. I worked a bit with zio, which is great. One of my long term side projects is a reimplementation of Nats (a message broker/communication framework), same vision, but focused on performance and lean resource usage and without the spaghetti code. One of the requirements is that such a system must handle a lot of connections and a lot of messages, which translates to a lot of Go routines, fibers, callbacks or whatever is responding to IO.

This is where you cannot just use async or concurrent whenever you have to wait for IO to do something, because every waiter get’s a stack, something presized from a few kB to MB. Small sizes lead to stack overflows or memcpy or various other issues that each have their price to avoid. Big sizes lead to massive amounts of wasted memory or if you are fancy at least address space.

It’s easy enough to write a custom event loop for use cases where you cannot use std.Io naively, but the problem keeps surfacing in other areas. For example, you need to parse a message header, but the header is not guaranteed to come in as contiguous memory area. You want to read the header before the message is complete, because the message routing is complex and takes time and why let other cores sleep while they have work to do. You don’t know whether the header is completely read until you start parsing it and hit the end of buffer. So you have to handle the case when you need to wait for more data.

Parsing a header requires access to the data. If the data is in cache, it’s there, if not, loading it takes much longer than the parsing itself. So you want to do the parsing in the thread that does the IO, because the data is then still in cache (depending on stuff). But if you do compute in a thread that does a lot of IO, everybody else has to wait for the compute, so that better be fast, really fast.

My header scanner uses simd and compared with a byte-by-byte scanner, it’s about 1.5-3x faster. Mind you, these a nano-seconds, not a big deal. But there is a reason why simdjson has such a great reputation. The scanner implemented using std tokenizers is 2-4x slower than the b-b-b scanner, despite also using simd, it just uses a fresh register load for every token you scan.

This scanner I wrote is surprisingly cool. I plan to use it for other things, given how convenient and fast it is. Like CSV, passwd, procfs stuff scanning. All of these might be used in performance critical contexts and all these contexts have to somehow handle fragmented input. You really don’t want to load an entire file into memory if you don’t know how big it is. Fragmented input means you need to pause scanning until you have data to scan.

When using std.Io, this is no issue, you just use a reader-kind of abstraction for your input and code away. If your input is not yet there, your scanner gets suspended and work continues once data is available and the core doing the work is doing other stuff. And this is not expensive, not much. You’re still doing high performance scanning, it’s the fault of IO that you wait. But, if you can’t use std.Io because the overhead for fibers or threads is too much for your use case, this is not an option. Then the scanner has to return something like EGAIN and somehow has to know how to proceed once data is available.

Here is the real problem: If you’re writing some library, you don’t want to write two versions, one colorblind and one with suspension. That’s the whole point of std.Io’s async/conc, getting rid of colors. But if colorless can, as a matter of principle, never get top performance because of the framework overhead that is immanent to any std.Io implementation, then you need to provide an escape hatch.

Zig without async can of course provide this escape hatch. You basically do what you would do in C, you write a state machine that allows resuming the code waiting for IO. Getting this right however can be rather hard. And it’s hard for each and every library that has to do it. That’s why people love async keywords so much, even if it means you have to write two versions of the same logic. It’s much easier to write correct state machines.

Most people will just use std.Io abstractions and it will be perfectly fine. Most library authors will do the same and it helps a lot to write performant and maintainable code. So what am I complaining about?

Zig is the language I would always choose - as a matter of first principles and concepts - if I want the best performance and the most control. This is what Zig does. Together with comptime, I can build zero cost abstractions. The set of design decisions in Zig so far is unique in this regard. There is no GC, no runtime, no frameworks. Zig has enough sugar to make it surprisingly easy and safe.

That is, until std.Io. This is not Zig the language, it’s just a library and you don’t even have to use it. You can implement std.Io yourself and tweak it. But no normal developer would ever do that under any normal circumstances. It’s hard.

Zig, compared to other languages, has a couple of disadvantages. A lot of functionality that is included in other ecosystems is not in the std library and also not yet provided by the community. The fact that the language is still evolving puts a maintenance burden on devs. That’s fine with me, I can wait, I can contribute, and it’s fun.

But what that means taken together is, that the ecosystem will evolve along the mainstream. Zig will eventually commit to std.Io and there will not be a sustainable path to write software with truly minimal resource usage and maximal performance. You can write optimal software, but then you have to write it all yourself. There will be no libraries that support stackless async, because to few users need it and everybody uses the one colorless interface. Even libraries that emphasize performance over everything else, will likely only support std.Io. And Zig will have a runtime. You can replace it, theoretically, but you likely won’t.

The thing I would like to discuss, after making this point above is this:

To implement my scanner in a reasonable way, I would provide a comptime parameter that has is something like either getMoreData(): u8 or getMoreData(): Future(u8). This parameter would then bleed into the function signatures of the methods providing scan results and also leak into some seams in the library. std.Io has a kind of future, zio does. But, to my knowlege they are either not public or somehow not really idiomatic Zig abstractions. There is no accepted Zig pattern for using futures, that’s the whole point of std.Io and removing async. So I would likely provide my own Future type and do my own state machine implementation, and so will everybody else.

This is not sustainable and it will in my estimation end up with people just paying the stack tax, Zig will have a runtime and it will be somewhere between excellent and good enough for most requirements and not the right tool if what you really want is both control over everything AND convenience.

With this particular experience, I get the impression that providing std.Io was THE RIGHT thing to do, but removing async was a MISTAKE. I think a language like Zig needs both, a colored async and a colorblind std.Io, unless colorblind and stackless are not mutually exclusive attributes, which they seem to be.

Your thoughts? Am I missing something?

4 Likes

I need to re-read more carefully, but one thing, you can configure zio’s initially stack size and still not risk overflow. The only reason the default is currently at 256KiB is because Zig’s stdlib routinely allocates large buffers on stack, so you actually need it in many cases, especially dealing with file names, and especially on Windows. Stackless would transform that into routine 64-256KiB frame allocations on heap and I’m not sure if that’s better. I personally started to appreciate having the stack space available, it turns into much faster programs in my experience, because many heap allocations can be avoided.

Additionally, if you accept some hybrid node, you could have implemented the network layer using zio APIs, where you are able to wait on multiple things without separate tasks, and handle the logic using std.Io APIs, where task per operation is acceptable.

Or you could construct two zio runtimes, one initial stack size of 2KiB purely for networking, one for processing, where the initial stack size is larger. Note these are just initial sizes, the stack can grow, risk of overflow is small.

For context: My issue is not practical, it’s probably more dogmatic than I like to admit.

I wrote a Nats client with python, the result was 0% CPU usage and ~40MB memory. The app is idling around most of the time. But it was fast enough and I have 96G of ram, so there really was no problem. It just bugged me so see this for an app that does almost nothing.

I ported the thing to Zig using the “official” zig client for Nats, and the result was 64MB memory and 170% cpu while the thing idled. It also used 6-7 threads. That is not Zigs fault of course, and the guys developing the nats zig client are not incompetent either. They just didn’t use the lib in production yet.

I forked the lib and tuned some parameters, got to 20MB memory and 0%CPU when idle, still 6-7 threads. You could count that as success, but I still hated the result. It’s better than using the native Go client, because that has this roughly 20MB overhead for its runtime and stuff, but Zig has no runtime.

Then I took a close look at the lib code and saw it uses the threaded std.io, apparently because std has no functional network support for evented. I found zio, and hacked up the lib to use it. Result: 7MB, 0CPU and 3 threads (numbers prone to memory corruption). That’s a huge improvement over what I had before and I really could have stopped there. But I just didn’t believe that this is actually what a mostly idle app listening on a domain socket should need to send and recieve text messages. I already reduced stack sizes for coros and chose params I thought are appropriate for the client.

So I took a deeper look at zio and the lib and looking at what they do, I am pretty certain that getting this app to 2MB should be easy. Zio spends a lot of effort doing things such a client never needs. There is no way to comptime that out, because it’s in the contract of an std.Io implementation.

This is all very focussed on memory usage, but that’s only because it’s easy to measure it. At runtime there are similar barriers resulting from zio’s concept (and that is true for any particular choice made by a GP std.Io implementation). The floor that a correct std.Io implementation can reach is determined by the contract. It needs a stack, it needs cancellation support, etc. Looking at what Zio does and likely has to do, this is more than a nats client has to do and maybe even more than what a nats server does for message routing and delivery (and that quite a lot of messy logic happening there).

My point is not that std.Io is not good, it’s excellent. So is zio, this is really good stuff. But so are Go routines, they are really well implemented and the concept provides a usable abstraction. I don’t like it, both emotionally and on grounds of principles. But I really don’t want Zig to become a compromise in terms of control and performance. Std.Io is a great choice for almost all use cases, but if it’s the only practical choice, it’s a limitation that I find painful. I really believe the stackless approach deserves an idiomatic representation in Zig world, because it solves a problem that std.io probably cannot solve.

Zio libraries (ev and more) are good, but not good enough to provide a substantial advantage over directly coding io-uring stuff, I don’t need portability for my use case.

As far as I can tell, std.Io in fact wants to provide lower level API as well, it’s std.Io.Batch, but it’s unfinished as of right now. Unfortunately, it lacks per-Operation cancelation, which is a must for e.g. servers.

Other point is that while it’s fairly low level, it still has to do some things implicitly, for example: plumbing between std.Io structures and OS structures: iovec/msghdr/etc. It just has to allocate them somewhere.

Zio’s ev module is a solution for missing implementations in std and I don’t mind waiting for std to evolve.

But the problem is not even so much the IO side of things than the effect it has on code that depends on IO.

Std.io provides the colorless abstraction in such a cool way that you simply write code as if it was sync and it works with async and concurrent. That’s amazing and you really don’t want to write two versions of the same code to support futures.

But once you commit to that, you also commit to the price of stackful routines. You can write your own eventloop, but you still have the to do something about how consumers of input handle fragmented inputs. The easiest way to do that is to use a reader and synchronous code, std.Io can run that async or concurrent without you having to worry about it.

If my scanner was a general purpose library, I would of course have used std.Io as input abstraction.

And that would kill a lot of the advantages my custom event loop would have. I had to serialize receiving a message before I can start parsing it, or I start parsing it, hit a premature-eof and then try again next time I receive data on that channel. But then I need an extra abstraction for “a message is incoming and incomplete, defer processing until more data arrives”.

At this point I had to choose between increasing the complexity of my network module (something that is already quite complex) or rewriting a scanner (assuming it would exist using std.Io). Not nice. And the same effect hits me with everything that consumes input or to a lesser extend produces output.

To me this seems to be a systemic problem that is at the same time very niche and fringe but really painful when it hits you.

I’m probably the no. 1 complainer std.Io here on Ziggit, but I want it to succeed, because I see compatibility across the ecosystem as the top priority. The interface still needs work to cover all the use cases, but I don’t think it makes sense to write networking libraries that use custom APIs at this point. For the first time, you can use one lib for HTTP server, one lib for Redis client and one lib for PostgreSQL client and and they will all cooperste under one event loop if you use an evented Io implementation. I think that’s awesome and it’s a huge value of Zig compared to languages like C/C++ where no such compatibility exists.

Regarding NATS, when I saw the official client, I was horrified. It makes me want to spend some time upgrading my own NATS client to std.Io.

Let’s do it together, I’m working on my own client (same conclusion).

Here is the fork I was working on, but I really don’t like hacking up stuff like this: GitHub - mutech/nats.zig: Zig Client for NATS · GitHub

Sadly the server side Go implementation of Nats is no better. Nats is a product, not a project. That bleeds through everything. Really sad, because their vision/concept is positively awesome.