Zig fetch EndOfStream error

korke · August 30, 2024, 12:18pm

zig fetch --save="x86_64-windows-msvc-llvm-18.1.7" https://github.com/llvm/llvm-project/releases/download/llvmorg-18.1.7/clang+llvm-18.1.7-x86_64-pc-windows-msvc.tar.xz

this command results in error: unable to unpack tarball to temporary directory: EndOfStream

so, was wondering if this is a known issue. or anyone else is encountering it.

Sze · August 30, 2024, 12:52pm

My guess would be that the temporary directory is hitting a physical size limit, for example being on a partition that is running out of memory while the unpacking is being done.

I don’t know where this temporary directory is located on windows.
I imagine if you have a system and user partition and the temporary directory happens to be part of the system partition maybe it is almost full?

All of this are just my first guesses, to dig into it more deeply it would be helpful to get a full stacktrace, I am unsure whether a normal Zig build gives you that, or if you require a debug build of Zig.

Do you have any more info? (Like operating system, zig version, etc.)

Have you checked whether any of your partitions are filled to the brim, so that they can’t fit the uncompressed archive?

korke · August 30, 2024, 1:42pm

Using zig-windows-x86_64-0.14.0-dev.1359+e9a00ba7f, space shouldn’t be an issue also tried on a linux vps, with the same version 0.14.0-dev.1359+e9a00ba7f, got same result, may be some issue from github’s end but manual download using python’s requests module and combination of tar xf command seems to be working normally

Sze · August 30, 2024, 1:53pm

I just ran the command on linux with zig 0.13.0 and got the same error.
I wonder whether it would be possible to enable a more verbose log level.

Maybe somebody with a debug build of zig can reproduce and see why it fails.

A normal build of LLVM+Zig already takes a long time on my laptop, so I am hesitant going for a debug build.

squeek502 · August 31, 2024, 1:04am

I believe it’s the download that is failing/being truncated, and Zig doesn’t retry on error. Relevant issues:

github.com/ziglang/zig

zig package manager should retry fetching on temporary errors

opened 04:52PM - 10 Oct 23 UTC

mitchellh

enhancement zig build system

### Zig Version 0.12.0-dev.790+ad6f8e3a5 ### Steps to Reproduce and Observ…ed Behavior My project (Ghostty) has ~20 dependencies it has to fetch from the internet given an uncached build. These are fetched from multiple sources (github, gitlab, self-hosted, etc.). Each push results in ~7 separate CI runs which run uncached. As a result, each Git push results in ~140 new HTTP fetches for these large files. Even with this _relatively_ small number, its big enough that I'm regularly seeing temporary errors fail CI. These are the sorts of errors that an immediate retry fixes the issue 90% of the time, and a very short backoff fixes the issue the remaining 10% of the time. So far, none of the failures have been real persisted outages. ### Expected Behavior Zig should retry on fetch errors. The ones I've seen but this likely isn't exhaustive: - `error.TemporaryNameServerFailure` temporary is even in the name! - HTTP 500 errors, namely 502 which is usually a single backend down while others are still up - HTTP 400 errors even, I've received a 404 before when the package existed, likely a CDN issue where some edges have it and others don't My recommendation would be to do a _short_ series of backoffs. Even backing off a few times with a maximum total wait time of 10 seconds would probably fix 90% of my issues, but perhaps this can be configurable because for CI I'd happily configure this to upwards of a few minutes. The backoff should probably increase, whether it be exponential, fibbonacci, whatever. **As a workaround**, I can make my whole CI job retry, but the CI job does a lot of other setup and no other steps in the job are flaky so this would probably result in retrying genuine failures and cost me more money. It'd be much more convenient and faster if Zig retried.

(downloading the file from the OP in Firefox failed at around 40% for me, but resuming afterwards worked)