Feature or Bug w/ `zig fetch --save`

For the past few months I’ve been using zig fetch on the master branch. The common guidance with adding dependencies is to use a url like https://github.com/[user]/archive/[commit-sha].tar.gz in the build.zig.zon and with the zig fetch command to get the hash. Using zig fetch --save is a cool step forward which allows us to forgo manually adding dependencies to the build.zig.zon as long as the project being added has a build.zig.zon of it’s own.

While using this command to add my own GitHub projects as dependencies, I’ve realized I can use https://github.com/[user]/archive/[branch-name].tar.gz instead of specifying an exact commit. This has a few benefits to me:

  1. It’s much easier to read and write.
  2. Re-running zig fetch --save on the same branch name will update the hash in build.zig.zon to the newest commit of that branch if there is one.
  3. Because of 2, I can separate different versions of my projects into their own branches (such as nightly and stable) and adjust my installation instructions to simply point to the proper branch. This allows users to update dependencies on my projects easily as seen above.

That said, I had a discussion on Discord a few weeks back with Sinon, who provided good feedback on why this might not be a good idea and could actually be a potential bug. I’ve also only tested this on GitHub projects, so I’m not sure if it’s consistent across other sites. Before I adjust the installation instructions in my projects, I’d like to get the community’s take on this. Is it a nice feature or an ostensibly beneficial bug?

Are you referring to this specifically?:

Or that zig fetch --save will update the hash?

I wasn’t aware that you could use the branch name, but in combination with the zig fetch behavior, this seems like a nice workflow to me, however I haven’t thought in depth about it and my experience with managing dependencies is still limited to very few manually managed ones.

After reading the linked discussion a bit and the mentioned github issue, I think there are two separate things here, the github issue is about avoiding copied hashes being misidentified as belonging to another url and just seems like a bug that is somewhat unrelated, while your question is more about developer workflow and best practices.

1 Like

You can also use git tags and that’s what I’m using in my projects. That allows me to freeze at a specific commit by tagging it and whoever adds the dependency based on that tag will be assured that they can get a reproducible build. My projects are on Codeberg.org and it works fine there as it did on GitHub.

1 Like

Good question! Originally, I was curious about both but now that I think about it the behavior I’m most concerned with is the zig fetch --save updating the hash.

I also agree that the listed issue in that discussion is separate from what I’m asking about.

To clarify the rest of this discussion, the focus is on zig fetch --save updating the hash for GitHub urls pointing to a branch name.

This is a good alternative, but I don’t think it will allow me to maintain a single url for a version that automatically points to the newest commit. Maybe that can be automated somehow? It’s definitely worth looking into if my current method is deemed wrong.

Thinking about it I think there is an issue with it, the issue is that your package contains urls that are associated with hashes, but the urls you are using change their content, meaning everytime you push to one of those refered to branches, you invalidate all previously released versions of your package and everyone who depends on your package is now forced to update their package too, because their packages won’t build anymore, because they will download new code that doesn’t match their old hashes. So basically all the urls used should always point to the same content and if the hash changes the url should change too.

To clarify I don’t think that zig fetch is the issue here, the issue is using urls that change their content in some unknown future time, that means that you only can depend on the latest version, but some users may want to keep using an old version.
Another thing is that those users will have a hard time figuring out which branch was updated causing the hash to change, now they have to do detective work to figure out what commit their version originally referred to. So maybe actually there should be a warning when you are using urls that aren’t based on a content hash, but I guess that is difficult to detect and site specific.

You’re right, the dependency would be forever tied to the specific commit that the tag points to. This is what some people want to guarantee they get reproducible builds. It’s like what the lock file does in other languages’ package managers. But if a suer just wants to follow a branch with the latest commits, then specifying the branch tarball and using zig fetch --save seems like a perfect solution to me.

What I’m doing is creating a tag when a stable Zig verion comes out, say v0.11.0 and tell users wanting to work with that Zig version to use that tag or subsequent tags in the line, for example v0.11.1 if an update arises. And then I tell users who use the current Zig dev version to use the main branch tarball as a dependency and update the hash when they want to move up to the latest commit. Not sure if this is the best strategy but it’s worked pretty well for the time being.

1 Like

So I guess to get the best of both worlds, we would need a feature that allows us to specify a branch and then some tool would automatically update url to point to the latest commit, that way you always have a url that matches the hash, but also can avoid having to update the url yourself.

So I guess we need to implement or wait for what is described here: ability to fetch dependencies via git+http and git+https protocol · Issue #14298 · ziglang/zig · GitHub

In that issue it seems that a fetch plugin should try to implement git + branch logic and I guess that plugin would need to be able to set the url to the latest commit.

Until somebody has implemented something like that I think it would be best to update urls manually to point to a specific commit, or build yourself a custom tool that looks up the commit from the branch and updates the url for you. I don’t know how much of this tooling will end up being builtin to zig, or part of some plugin.

Regarding the branching information I wonder whether we could have an optional field in a dependency that specifies the branch, then custom tools could look up that branch check if there are new commits, ask the user if it should be updated and then update the zon file to have the new url. That way zig fetch wouldn’t need git support which seems to be like its unwanted, when you consider this comment ability to fetch dependencies via git+http and git+https protocol · Issue #14298 · ziglang/zig · GitHub , the ideal case seems to be that zig doesn’t know or care what git is (to keep tooling minimal, functional and non feature creep) and then custom tooling or plugins can deal with those extra things. But I haven’t found anything official about where it is going.

Have a look at those issues if you want to investigate more: Package Manager · GitHub

1 Like

The use-case of checking for updates is mentioned here: add tooling to deal with build.zig dependency trees · Issue #14288 · ziglang/zig · GitHub

As you noted, though, implementing this for URLs involving a commit ID would require the branch to be specified somewhere as well. On top of that, the GitHub archive URL format is specific to GitHub, so updating such archive URLs would require host-specific logic to “parse” the archive URL and reconstruct a new one (this particular problem would be avoided by using git+https dependencies).

If anyone is curious as to how this might be implemented, it is straightforward to find the current commit ID for any branch in the repository using the Git support currently written for Zig: zig/src/Package/Fetch.zig at bb0f7d55e8c50e379fa9bdcb8758d89d08e0cc1f · ziglang/zig · GitHub (set up a git.Session, use session.discoverCapabilities to initialize the session, and then look for the branch using session.listRefs). The git.zig file implementing this support could potentially be copied into a third-party “zig-update” utility or something to explore the use-case and find a good design for it.

3 Likes

Appreciate everyone’s insight on this. For the time being, I’ll add the branch url as a secondary installation method and be sure to mention these caveats (probably with a link to this discussion).

Down the road, I think I’ll try my hand at creating a zig-update type utility that can hopefully be turned into a fetch plugin if those become available.