Build.zig.zon: error: unable to unpack git files: PathAlreadyExists

I’m currently working on a hobby OS (which I may also post about if I ever finish getting to userland), and I’ve been running into build issues on CI only on macos and windows (which I can only replicate on github CI since I work on this project mostly on my linux box).

I pull in various projects through the package manager in my build.zig.zon (as part of a long term goal to be able to drop in build+run on any target), and the build is failing with git being unable to unpack these deps. The confusing part here is that these are all lazy dependencies so I would have though nothing should be pulled in anyway as my CI is set up to not use these .

Here’s an example of the unpacking failure

Very likely that there’s a pair of file paths that only differ in their casing (e.g. foo/bar and foo/BAR). Windows and MacOS typically have case-insensitive filesystems.

1 Like

Sure, but what does that amount to then? A zig fetch bug/defect? Or something I’m doing wrong (which I can’t see what that would be just yet)? All I’m doing is building with all LazyDependencies unused, I would think it would be up to the zig package manager to resolve directory existence/cache misses on its own

That’s quite a complicated build script. Are you running multiple zig build simultaneously? Because it looks like a race condition to me. lazyDependency is checking if the package is there, it isn’t, then it tries to fetch it, but it fails because, in the meantime, someone already put the package there. If I’m correct, rerunning the script now that the package is already fetched should work.

Zig supports packages that contain files that may be unsupported by some file systems. This is necessarily true, because you could invent a file system tomorrow that forbids files whose sizes are prime numbers, and that doesn’t mean Zig suddenly has a bug that wasn’t there yesterday. It’s up to those creating packages to avoid tripping common file system pitfalls.

In more concrete terms, it’s a valid use case to use Zig to fetch packages that have files with names differing only in case, and to use those packages only on operating systems that support such things.

2 Likes

Nope, or at least this shouldn’t be (but github CI is well known for being ugly around the edges). This should just be running 4 sequential runs of the “kernel” step, each with slightly different build flags

1 Like

zig build supports being run simultaneously. The cache system uses advisory file locking to accomplish this.

Here we see since they’re both waiting on the same resource, one process does the work while the other waits for it.

3 Likes

So a few things:

  1. I am using tagged 0.16.0, not sure if that will change anything but it’s worth noting

2.) the offending package is one that I’ve put together myself, which is just a small repackaging of schillytools (very small subset just to be able to create iso images) and this is just c-code. I’m not sure how any filesystem hijinx could be at play

Edit: the schillytools fork for reference: GitHub - spineda2019/schilytools-zig · GitHub

Edit 2: 2 other big details:

  1. This only happens in my dev/userland branch, so any build script or zon details that matter would be there
  2. I dusted off my macbook and tried a normal zig build and I get the same issue, so it’s definitely replicable

Hi, FWIW I just cloned and ran the five build commands from the “Build Kernel & Docs for all targets” section in the CI script (in the dev/spineda/userland branch) on a windows machine and they all seemed to work fine.
EDIT: no same issue

1 Like

Yeah it’s just what’s in my dev/userland branch, not main (which I’m assuming is what initially worked for you)

yeah, I was looking at the correct branch in the browser but did not switch locally :face_with_peeking_eye:

You have to call lazyDependency only when it is actually needed. First sentence in the docs:

When this function is called, it means that the current build does, in fact, require this dependency.

Right, which is what is done, unless I am missing something. Every lazy dep is guarded with a user passed build flag. I only default to using them when moving towards a stable feature like with building bochs, but my CI is setup to not use any deps at all

This line is run regardless of the option, right?

Edit: you could move below this if-statement:

1 Like

Huh, yeah I guess I forgot to guard those (I changed this in this small commit)

My point stands though that I can’t tell why this unpacking is failing though. The schilytools fork I made doesn’t even have crazy stuff like symlinks, it should be pure source

I am going to give zig master a try to see if any difference is made, but that would be slightly unfortunate since I want CI to use stable tagged versions

Plus building schilytools itself with zig on linux on CI works as well: hash fixes · spineda2019/schilytools-zig@05acf32 · GitHub.

Not sure why the unpacking is failing on windows/mac (again, just the unpacking. I expect schilytools to not be build-able since I’m not done with the mac or windows port)

As others said before, schilytools likely contains some paths that only differ in casing. You can write a script and search for those on your linux machine.

1 Like