I’m opening this thread to re-route a discussion here about Software dependencies - @JPL brought up the following point and we can continue the discussion here:
And also added:
So, what’s your opinion on the matter? Should Zig projects strive to have less dependencies overall or should integration be a more common part of Zig projects?
I’d say language purisim is not as important with Zig projects as it is for other languages. Zig brings it’s own C/C++ compiler along with it. So if your dependencies are managed fully by Zig (ie. fetched and built) it makes little difference if they are also written in Zig or not. Although, I think this applies only to large dependencies that are not worth re-writing in Zig. The Lua language VM for example. Or perhaps tree-sitter, or ffmpeg.
I think generally Zig projects should strive as much as possible to build entirely independently of the host OS, whether they have C/C++ dependencies or not. That is not always possible (OpenGL libs on Linux for example), but it’s really, really nice when it is.
This is a topic that periodically pos up in the back of my mind and I still don’t know what the best option should be. But I’ll just drop some of my reasoning here for discussion.
First thing that comes to mind is the whole leftpad crisis where a whole ecosystem of software can be crippled if just one dependency is suddenly removed. Kinda like Jenga blocks. It’s not that you never ever use dependencies and always reinvent the wheel, but it seems there has to be a “reasonable threshold of complexity” where you can decide not to depend on a third party and just code it yourself.
Then there’s the impact it can have on the actual software development experience. Take Rust as an example. A hello world project in Rust will compile pretty fast. Not as fast as Go because LLVM is involved, but nothing to really hurt the dev experience. But a typical Rust project these days has a ton of dependencies (try compiling actix-web for example) and then the compile times just keep adding up. I believe this is the reason why Rust has gotten a bad reputation with regards to compile time.
In the case of JavaScript and Node.js, compile time isn’t an issue but here the bad reputation revolves around “downloading the Internet” (node_modules) just for a simple project build.
Complex dependency graphs will also require increased complexity in the tools needed to manga them: package manager, build system, language server, etc. So that’s another tradeoff.
I think we should always strive for fewer dependencies.
That makes ourselves more accountable for our software and makes us really think about what’s essential and what isn’t.
I have an unpopular opinion here: but the fact that it’s relatively hard to manage dependencies in C or C++ projects is a good thing: it makes me think twice before adding one.
A great example of this is the Kakoune code editor, which has no dependencies, not even ncurses, or a regex library (it used to, but gradually reimplemented the needed features).
In other languages, adding a dependency is so easy, that when building a project which depends on B and C, but both B and C depend on D, but with a different versions, we end up compiling D many times. I won’t name names, but if you’ve used Rust, you know what I’m talking about (I know this is because everything is statically built, too).
Still, it’s amazing how the simplest programs have hundreds of dependencies.
As an example, projects which serve similar goals, rust-analyzer and zls, have 38 and 2 dependencies, respectively.
We should strive for simplicity, not ease. That’s more aligned with the mantra of Zig, which is moving into that direction.
And for those who think we should not keep reinventing the wheel, this is what we would have if we hadn’t reinvented the wheel over and over.
Isn’t the OS and hardware a dependency?
The problem with libraries is that they don’t have the quality level of OS and hardware dependencies or there are no resources to ensure that a library is of the required quality.
IMO this is one of the hard problems that if solved would really propel software engineering to a new level. Rust’s #![forbid(unsafe_code)] comes to mind, although not strictly a software quality measure, it’s an example of how we could try to impose some restrictions on how the language is used in our projects. Who knows, maybe AI could help in analyzing and detecting violations of software quality expectations? Establishing what software quality is would be a whole other ball game. Perhaps that should be left as a project-by-project level decision.
Zig’s biggest benefit is seamless(most of the time) interoperability with C.
I think a lot of it also depends on how successful Zig becomes. Right now what sucks the most is the fact that a lot of dependencies are written in foreign languages which don’t offer the same guarantees; currently, there is a lack of options in Zig to replace some very useful libraries that one would need to depend upon for building anything in a reasonable timeframe. Which is normal; the language is young. But I think that in the future if Zig truly replaces C and people are motivated enough to rewrite stuff in Zig, then I don’t think that dependencies will be such a big issue. Especially considering all the thought that has been poured into the build system to be user/dev/system package manager-friendly.
I think a cool option would be to create a community-based MOST WANTED list. You list what dependencies are the most used or wanted by the community. I don’t know, for example, ncurses, then you gather people motivated from the community, and start working on it together or if some Zig projects already exist, you contact the maintainer, and you bring the motivated people from the community to help speed up the development. And little by little, the community starts to rewrite some very useful libraries.
Although that doesn’t solve the problem of dependencies, it greatly reduces the burden for our community but also for others. Like one could imagine that if we have a cross-platform, Zig-based equivalent of what used to be a C/C++ library, that a lot of people who want to build something using such libraries would be pleased to know that there is an existing alternative.
It also helps us reinvent the wheel in a more focused and united ways. My Adjutant in the army had a saying for that “alone you can go faster, but together you will go further”.
Few to no dependencies is nice – when that realistically makes sense and isn’t too much of a burden on you and people around you, from now and into the future.
Some recent personal experiences that support my die-hard centered stance on this:
For a work project I implemented a “CSV” parser that reads a single huge file from a know generator with horribly buggy output. It was easier to just do it myself than to convince a generic parser to do it. The code is short, self-contained, the data format isn’t subject to change, and thus I expect nobody will ever need to work with this code after me to extend it or whatever.
For another work project I needed to generate XLSX files with various types of data, with more expected in the future. Of course I introduced a dependency to an existing approach to do it, because 1) I personally don’t care the least bit about reinventing the wheel of Excel file generation, and there’s plenty of solutions that do a good to great enough job, 2) it’d probably take me months to get to understand enough of the spec to be able to output what I needed, 3) and if I did that I’d be signing up for a jobtime of maintainership for a thing I absolutely don’t care about.
So yea, context matters, a lot. I don’t know who Zig should aim to please but it’s probably safer to go the purist way (there’s slightly less chance of doing the wrong thing if you don’t do things at all).
I would put this slightly differently: Zig’s biggest benefit is making it trivial to provide a C compatible ABI for Zig libraries. This is possible in any systems language, but Zig goes further than the rest, its program model is very close to C, and it can compile C dependencies as well as providing the right ABI surface and headers etc.
This is a boon for interoperating with C, but more than that: it allows Zig libraries to be used in any language with an FFI. That’s what drew me to the language, the ability to write libraries which can be integrated with any other language, with the right bindings.
I’m curious, in fact, to what degree comptime can be used to easily generate e.g. Python bindings for a library. If anyone wants to start a repo to work on that kind of thing, post it here for sure, I’d like to help out.
As for dependencies in Zig projects, it’s inevitable that as the community grows, we’ll see some maximalist projects which import the whole world from a deep dependency tree. Culture matters, though, and Zig culture emphasizes minimal dependencies and complete solutions. This naturally leads to self-contained best-in-class libraries with shallow dependency trees.
But not zero dependencies, unless that happens to make sense. It’s silly to roll your own regex just to have it in your program, and dangerous to do that sort of thing with cryptography. So I’d expect those sorts of functionality to be provided as libraries, with the community naturally gravitating toward the best-in-class.
It’s worth noting that Go culture emphasizes minimal dependencies, and shallow trees, while Rust is at the opposite end. Those aren’t language features, they’re features of the communities which use the languages. The kind of people who are drawn to Zig don’t want their packages to depend on deep complex trees which have transitive dependencies they don’t even know are there. So Zig code will tend not to have those.
This is what ChatGPT says, and I agree, and I would even add that I have seen a good number of projects (even cited in lists) obsolete and I’m not even talking about GitHub or anything else.
my answer:
I agree and I’m even more critical than you, because with dependencies, we often see a lot of mixing of languages, inappropriate functions with hard-to-maintain hardcoded code, and a lack of tools to maintain coherence. I would also add the lack of logical coherence in project construction.
chatGpt’s answer:
It sounds like you have a keen eye for the challenges and pitfalls associated with dependencies in projects. Indeed, mixing languages, inappropriate functions, hardcoded and unmaintainable code, and a lack of tools for ensuring consistency can all exacerbate these challenges. Additionally, the absence of coherent logic in project construction can further compound these issues. These are all crucial points to consider when discussing the implications of dependencies. Would you like to incorporate these points into your discussion as well? If so, I can help you refine the language and structure for clarity and impact.
There’s an aspect of dependencies that I don’t think is being discussed: are you working with a proprietary backend?
I took a break from writing Metaphor and I came to the conclusion that I’m going to start using the BLAS libraries much more than I currently am. I wasn’t convinced by the performance or difficulty arguments (I’m a hardcore DIY person at heart).
The thing that finally sold me was the fact that all of the code I was working on will only be applicable to one hardware manufacturer’s devices. Those devices, btw, can change and invalidate my code.
This also restricts me (timewise) from being able to support other hardware with the same frontend.
If accepting a dependency allows you to skip over a lot of proprietary ugliness and uncertainty, it definitely can be worth it.
I’ve worked my entire career with IBM, and I’ve been part of a validation committee for LG3 and LG4.
As a systems engineer (former term for man system) and project manager"
This is just an exchange of views, there is no personal attack.
I have the habit of exchanging by ignoring feelings and leaving room only for logic (for IT)
I feel like with Zig as it is now, the lack of a centralized package listing does encourage people to reinvent the wheel more, or at least copy-paste and maintain their own dependencies. It reminds me of other open source projects in C/C++, where there isn’t a standardized package manager. This might actually be a good thing, because less dependencies means that it’s easier to understand how each piece of a program fits together without any unnecessary cruft. I’m not saying that you should rewrite a whole web engine just to render a web page for instance, but having less dependencies means that it forces you to prioritize which dependency to put in a project (on a spectrum of one large dependency vs. hundreds of smaller dependencies). Just my two cents.
