There should be an API for creating archives (tar, tar.gz) in std.build.
Why:
We should not rely on code distribution platforms (github, codeberg, etc) to create code archives for releases. It’s important for zig to own the full process of distributing code to reduce our dependencies.
It’s non-trivial to create build-artifact quality archives. Some thorns are: file permissions, file names (cross platform stuff), file modification times (binary reproducibilty)
The std.build API could provide sensible defaults that aid our use case of redistributing code / build artifacts like encouraging reproducibilty even across platforms.
One complication here is that, as far as I understand, there’s no “good” archive format: different operating systems prefer different formats, and formats themselves are piles of accidental complexity, which make security very hard.
It almost feels like it could be a good idea to invent some kind of Zig Archive file format?
I don’t think that would be worth it because who would actually use it?
It will be worse than MSIs, deb, rpm and whatever macOS uses for the purpose of OS integration.
It won’t really be able to give you anything over zip and tar when it comes to simple release ball, but ensures that users will need a new tool to actually extract things, and is without the guarantee that you won’t end up with a lot of complexity yourself.
But as CPack has shown, if you are willing to put in enough effort, you can actually abstract things over a LOT of formats.
But even then if you don’t want to put in that effort, having simple binary archives based on the zip, tar and e.g. gzip implementation in the standard library which are needed to the package manager anyway, should be too far fetched.
Maybe this could even be extended with some form of plugin interface for other formats.
I realize it’s unrelated to this thread but I can’t help bringing attention to that absolutely ridiculous default in the cpack docs you linked. That’s what you get when the only variable type you have is string.
The default setting is as follows: "/CVS/;/\\\\\\\\.svn/;\\\\\\\\.swp$;\\\\\\\\.#;/#"
Sounds like a linux move. Releases are in the DevOps domain and have nothing to do with zig’s std. And you don’t need anything else in std to remove dependencies—just use the current zig to write all required tooling. Plus, your pipeline will run on third-party software just as well (operating system, in this case). This rabbit hole is endless (unless you have resources to print in-house hardware too).
We could certainly design a format using some limited subset of SQLite(a format recommended for datasets by the Library of Congress), but tar seems fine and popular?
What features are missing or should be deleted from tar in your opinion?
Now that I’m reading up on SQLite more, its becoming more appealing, since they even recommend it to be used as a data transfer format: Appropriate Uses For SQLite and there is even a pre-specified Archive file format definition: SQLite Archive Files
But including thousands of lines of C in the zig repo is probably a non-starter.
I’m not sure if that has been considered by the team, but I won’t be surprised if alongside the package torrenting proposal, we get some additional goodies like declaring mirrors for packages
Thank you for sharing this, the ZIP writing procedure itself is really helpful, I’ve being trying to implement a zip writer in the past myself but gave up because of how complex it feels. But also, the Shell thing in a whole is really fascinating.