Should the standard library be "batteries included"?

Brainstorming
41

Personally, I don’t think you should try too hard on that front.

From my point of view, stdlib should be adequate and portable and stop. Sure, you don’t want stdlib to suck; having most people using the same ERRNO and Posix abstractions is a good thing, for example. However, the amount of effort to move the needle from 85% to 98% may be an exponential amount of effort. Trying to abstract over Posix and Windows threading models, for example, is fraught with complications and corner cases.

Zig is a systems programming language. If you are using Zig, there is probably something that you are doing that will not fit the stdlib abstractions or performance guarantees. That’s both fine and expected. If you didn’t have some pain point, you would be using something garbage collected instead of Zig.

And, if you want to program with a zillion layers of impenetrable abstractions and dependencies, Rust is quite happy to accommodate you. Ganbatte.

13 Likes
42

Like C++'s STL, with some decent containers and algorithms, you got yourself a really good std lib.
With ArrayList, HashMap (and variants), and BoundedArray ( :headstone:), you’re set for containers. Binary search, lower/higher range and sorting cover most of the algorithms. That’s the 85%.

43

it may, but isn’t it more interesting, in the face of “this might be a lot of work” to go “well, i do do a lot of work all the time”?

44

To lend my voice to this conversation:

  1. I really appreciate the standard library in all its current glory. Words cannot fully express my appreciation. To know that most of what I need to write my program is available directly and I don’t have to import random libraries is amazing. For better maintainability though, I do think the tiered system suggested earlier is a good approach with blocks in each tier being evaluated for promotion when they meet certain requirements.

  2. I personally treat the /lib as part of my program. I don’t consider the standard library any different than I consider my own code. If I need something, I simply click into it, make changes, and it turns out zig compiles with my changes to std. Now, this isn’t recommended for forward compatibility. If I need to keep my change, I simply copy the said function into my code, but,
    a. having the standard library show me how to do certain things is amazing
    b. having the mental model that std isn’t a black box is refreshing.

  3. I understand why some people may be unhappy with Io, but I personally think the idea behind it is the best thing that has happened to the software industry since DOD. The new Io clearly addresses the latency of a program if you truly understand it, and I’m excited to see it get to completion. Pending that time though, it is my responsibility to write my own event-driven state machines when I need those.

  4. I see no reason why I won’t reach directly into Io.Theaded to call a function directly if it’s public or copy it into my program environment if it’s not. Zig has an amazing std.os interface, and most times, when I know my program will run on an x86_64 linux server, I simply use os.linux. There’s no reason why I have to be generic about it by accepting an io parameter in the target function.
    So, to me it’s perfectly fine that std.posix is going away.

7 Likes
45

Tbh I’m totally fine with the basic idea that the stdlib only contains stuff needed to build the compiler toolchain but nothing else - at least this defines a clear rule what goes into stdlib and what doesn’t. If the Zig toolchain needs tar support (probably for the dependency system), then it belongs into the stdlib, otherwise it belongs into a dependency.

Bonus points if the stdlib defines related standard interfaces which then can be implemented with third-party packages (e.g. a common archiver/compressor interface where the stdlib tar code is just one implementation). E.g. the stdlib might also be a place to establish common high-level patterns and interfaces, which can then be used as ‘glue types’ between 3rd-party packages.

I also think that it is a good idea to have a clear separation between ‘Zig the language’ and ‘Zig the stdlib’, e.g. it should always be possible to write useful Zig programs without or with only minimal stdlib usage, but without losing ‘language features’. This paragraph might sound strange to people who never worked in ‘modern C++’, where the clear distinction between language and stdlib has been watered down. It makes a lot of sense when coming from C though, where it is quite common to mostly ignore the stdlib (because it is mostly stuck in the K&R era).

9 Likes
46

I personally like what’s happened in the python ecosystem. The standard library is quite “batteries included” but there are still some functionality where it’s just better to use a package, the best example being unittest vs pytest, everyone I know just uses pytest.

One of the big issues with stdlib libraries specifically is the necessity to be forever backwards compatible, this usually leads to issues down the line as the abstractions end up not fitting the real world over time. Which is why having a thriving ecosystem of packages helps a language IMO, especially since non-stdlib packages allow for a lot more exploration, I feel like a language’s stdlib should be “safe”, as in, it should fit the common use cases but wouldn’t try to fit every use case.

4 Likes
47

Python is actually a great example. Python is obviously very high level, but their socket module is low level mapping of the Berkeley socket API, it’s flexible enough to be used in many contexts. Even asyncio streams use the socket module.

All my previous comments are not against Io, I’m very much for it. What I was saying that stdlib should have the lower level base for implementing higher level abstractions. The way it is right now, it prefers the one and only implementation, or do everything yourself from scratch.

48

I’ve said it before, but I think the long term role of any good standard library, is to be the foundation of the ecosystem, it should be filled to the brim with the right high impact bricks, Allocator interface, Random interface, Io interface, Reader/Writer interface, Data structure, iterator abstraction, cryptography, networking, file system, Basically a core from which the whole ecosystem can be built with an interop layer (to me a good std is kind of a language on top of a language it’s really important).

We can already see it in practice, if you fetch any Zig package, if it does memory allocation you can pick the allocator to fit your needs, if a package needs an Hash map, you don’t have to fetch FooBar’s Hashmap on github, you can just take the one in std.

Now the biggest advancement recently has been Io, having a way to control and schedule any library to fit your needs, is absolute godsend. I can’t begin to explain how frustrating it has been for me to see amazing industry standard C/C++ library, which would have been a perfect fit got to waste, because I couldn’t actually control the scheduling of operations, without needing to make a soft fork.

The one example that I have in mind is Libcamera, Libcamera is a library to use, configure, and manipulate cameras, isp, encoder/decoder, transform etc. It’s the industry standard, and it supports a ton of platform, android, raspberry, and many more, but because it supports a lot of platform, Libcamera is built around threads (with a Qt like signal layer), which on a tiny raspberry pi is super wasteful if you are already hammering the CPU with computer vision tasks.

At work I would have loved to use Libcamera, and in fact the project initially was using it, but because we couldn’t integrate it with the refactor of the whole application around an epoll loop, I had to code a worse, but more optimized “mini Libcamera”, built in C around epoll and v4l2, which was tremendously more efficient, and suited for my usecase, where readiness is indicated with dmafd polling, and h264/yuv readiness is indicated with eventfd, to dedicated worker thread.

This was a lot of work, and I had to reimplement everything, the runtime, the initialization, the capabilities negotiation, the ISP’s meta-data handling to implement color correction, Automatic Exposure, White balance, even parsing the Libcamera’s JSON tuning format, even if it was necessary for the goal set in my project, and I was successful.

I still consider this work a pure and complete waste of time (outside of the fact it was very interesting to learn and get nerd sniped into), because my code is objectively worse than the one in Libcamera, it’s not standard, it doesn’t have full feature parity, and I have to actively maintain it.

All of this would have been solved with the Io interface, because instead of being forced to make my code work around the decision of upstream, I could have fit upstream around my needs, and have it work around me not the opposite.

So to me I think the std, should focus on those building block, giving the ecosystem the foundation to build, and interoperate, to maximize flexibility and code reuse. And in my perfect world, there would also be a std-contrib, similar to opencv where opencv core is the well maintained, guarantee tier one code, and opencv-contrib is the experimental, often SOTA, but still too early code, that you can use, and rely on but without any guarantee for it’s long term support.

I would love for Zig to take that direction, separate what should be core in the std, from what shouldn’t be for example the http module, I’m not sure it’s necessary to be part of the final 1.0 std, but instead i would love to see http.zig be promoted to some kind of std-contrib, because it’s such a great project, and while http itself probably isn’t “core” some kind of “hey this is what we have elected to be a great baseline” from which anyone can build upon seems like the best of both world.

The core maintainers, focus on shaping the building blocks, with direct feedback from the std-contrib maintainers who got their package promoted/vendored. It makes it easier to pick the right package, and promotes code reuse.

11 Likes
50

Not mixing Io and protocols is a solid idea. You may like

51

I imagine that there was a reason for this that has nothing to do with the quote that you made. Maybe you should have a chat with the moderators in terms of getting on the same page as far as expectations for participation here, rather than putting random ominous warnings in unrelated threads.

15 Likes
52

At some point™ C++’ std::execution (yes, that got finally into C++26) will be rather well used by different projects since it is pretty similar in philosophy to Zig’s std.Io, even if it’s in typical C++ fashion overly complicated and with template hell.

Until then, yeah, things are going to be annoying.

3 Likes
53

Ok, but what happens if the compiler no longer needs tar support (unlikely, but it’s the example given). At some point (1.0) the standard library becomes the foundation of most Zig programs, and removing something from the standard library will break a lot of codebases.

In the Python world it’s a hard learnt lesson that “the standard library is where code goes to die”. Not because it becomes useless or bad, but because when everybody depends on it changes are almost impossible. Raymond Hettinger, one of the long standing standard library maintainers, would quite often advise people against submitting their “cool feature” for inclusion in the standard library. There’s numerous examples of things that were included in earlier years that were regretted later because they weren’t mature enough.

I think the std library will need to move away from “what the compiler needs” to “what are good universal building blocks” over time. There’s obviously overlap between those sets, but I expect some compiler dependencies to be moved out as they’ll want to be more “alive” than the std library can allow.

7 Likes
54

In the Python world it’s a hard learnt lesson that “the standard library is where code goes to die”.

Not in my experience. I’m using Python at work on a daily base since >20 years, and the standard library has been useful and solid and quite stable from the beginning, and in comparison to Java (and JS, of course), you can get a lot of work done with the standard lib and only a few extra libs, if at all.

For me, “batteries included” is as important as the language itself.

Andrew made a point about trust: You only need to trust a small circle of persons to use Zig. I think this is important for the standard library as well.

3 Likes
55

Same procedure as for any other deprecated feature: mark as deprecated and move into an external package, keep in the stdlib for a grace period, and then remove it (whether that ‘grace period’ means until the next major or minor version after 1.0 is up for discussion I guess)

In general I think it must be possible to remove features from the language and stdlib even after 1.0, that’s infinitely better than accumulating cruft only for the sake of backwards compatibility (see what happened to C++).

10 Likes
56

On the other side removing things from the language after 1.0 has the risk of what happened with the Python 2 → 3 transition, which effectively took way over a decade.

And if you ask the developers behind it, they regret it and wouldn’t have done it if they could have foreseen that.

57

No. The transition from Python 2 to Python 3 was a massive change of the language itself and that’s not really related to the standard lib.

I consider Py2 and Py3 as two quite similar languages. The implicit conversion between the types str and unicode based on ASCII encoding was a serious design flaw in Python 2, which caused an unnecessary amount of bugs in many languages (except English).

Porting code from Python 2 to 3 was some work for my code, and I still found some bugs (far from the happy path) months later, but, hey, now Python is much better and it was well worth the effort.

I’m still on Zig 0.15.2, and I reckon I’ll have to rewrite my Zig code a few times until “1.0”, but I expect it to be similar to what I had to do with Python, and I’m not afraid of it.

58

The point is that once Zig is at 1.0, most people will no longer tolerate changes like that. Plus my understanding is that Zig plans to provide the same guarantees as Go once it reaches 1.0 – full compatibility between releases.

59

Read floooh’s message again: “remove feature from the language and stdlib”

60

We are drifting slightly off-topic here. What I mean is:

Stability of the language and for the standard library should be a goal, but not a guarantee.

Changes are a part of evolution, and sometimes, in rare cases, after careful considering, breaking changes are needed.

There’s a reason why the major version number was bumped for Python 3, and I think the 2 to 3 (r)evolution is a positive example for when it’s necessary and at the same time a warning to not make breaking changes lightheartedly.

Coming back to the topic:

The main reason why this took so long was that everyone waited for everyone else with libs and frameworks.

And this probably would have been worse (or impossible) if there wasn’t the batteries included standard lib, I guess.