Zig fetch failing with "discover remote git server capabilities: CertificateBundleLoadFailure"

nurpax · February 16, 2025, 9:50am

I updated to zig dev 0.14.0-dev.3224+5ab511307 and looks like I can’t zig fetch anymore:

zig fetch --save git+https://github.com/natecraddock/ziglua 
error: unable to discover remote git server capabilities: CertificateBundleLoadFailure

I’m not sure if this is due to some macOS security restriction or something else. Zig fetch used to work before.

This also affects running zig build, for example, I can’t build ZLS anymore:

zig build
/Users/janne/dev/zls/build.zig.zon:16:20: error: unable to connect to server: CertificateBundleLoadFailure
            .url = "https://github.com/ziglibs/known-folders/archive/1cceeb70e77dec941a4178160ff6c8d05a74de6f.tar.gz",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/janne/dev/zls/build.zig.zon:20:20: error: unable to connect to server: CertificateBundleLoadFailure
            .url = "https://github.com/ziglibs/diffz/archive/ef45c00d655e5e40faf35afbbde81a1fa5ed7ffb.tar.gz",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/janne/dev/zls/build.zig.zon:24:20: error: unable to connect to server: CertificateBundleLoadFailure
            .url = "https://github.com/zigtools/zig-lsp-codegen/archive/e1f281f67ac2cb8c19d3cabe9cfae46fde691c56.tar.gz",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I’am on macOS 15.3 (24D60). I’m not sure which Zig dev version I had before but I think it was 2800’ish.

Could be I missed some macOS security prompt, but I also don’t see anything special under System Settings/Privacy & Security (where you’d normally add app specific exceptions for binary downloads).

squeek502 · February 16, 2025, 10:05am

This seems potentially relevant:

It seems possible that the intention was to read one or the other or both, but it might inadvertently have been implemented to fail if one can’t be read.

nurpax · February 16, 2025, 2:05pm

Thanks for digging into those issues - I subbed to the related GitHub issues.

It’s a pretty serious regression as it makes Zig useless for me. I guess not everyone in macOS is affected - otherwise I’d imagine there’d be more people noticing this.

floooh · February 16, 2025, 3:20pm

FWIW I just tested 0.14.0-dev.3224+5ab511307 on my Mac (macOS 15.3.1 (24D70)), and I cannot reproduce.

What I did:

delete the global zig cache
in GitHub - floooh/pacman.zig: Simple Pacman clone written in Zig., delete the local zig cache and output directory
zig build works and seems to the fetch the dependencies fine

A manual zig fetch also works.

AFAIK I haven’t changed anything on the macOS side that could be relevant.

I did consistently run into trouble with zig fetch on my Ubuntu laptop though, but this seems to be some incompatibility between Linux and my DSL modem acting as DNS Server:

github.com/ziglang/zig

error: TemporaryNameServerFailure when using package management on Termux

opened 09:37AM - 13 Feb 23 UTC

leap0x7b

bug standard library

### Zig Version 0.11.0-dev.1606+3c2a43fdc ### Steps to Reproduce and Obser…ved Behavior 1. Create a Zig project using either `zig init-lib` or `zig init-exe` 2. Create a build.zig.zon file: ```zig .{ .name = "viisi", .description = "A RISC-V hobby computer inspired by old 80s/90s UNIX workstations", .version = "0.1.0", .dependencies = .{ .clap = .{ .url = "https://github.com/Hejsil/zig-clap/archive/272d8e2088b2cae037349fb260dc05ec46bba422.tar.gz", }, }, } ``` 3. Run `zig build` ### Expected Behavior It should be able to resolve github.com and download the file.

…this was solved by changing the DNS server to Google’s 8.8.8.8 / 8.8.4.4.

PS: doing a zig fetch --save git+https://github.com/natecraddock/ziglua also works for me.

nurpax · February 16, 2025, 5:23pm

I can try do track down a zig-dev version right before and after when the macOS keychain changes went in and see if I can narrow this down to a smaller range of versions.

I can in fact git clone the pacman repository and zig build it just fine. But if I run a zig fetch (as above) in that directory, I get the same failure I got in the original post. I don’t want to blow away my global cache right now as that will render my zig malfunctional.

EDIT: uhh, I guess I won’t be narrowing this down yet as I don’t know where to find a list of nightly builds from the recent past.

ianprime0509 · February 16, 2025, 6:41pm

One way to get old master builds is to check the commit history of the www.ziglang.org site: Commits · ziglang/www.ziglang.org · GitHub All the “CI: update master branch builds” commits are to update the master branch download links.

In this case, it looks like the last one before the PR in question was merged was 0.14.0-dev.3091+42e48b83b: CI: update master branch builds · ziglang/www.ziglang.org@1eb513a · GitHub (see Comparing 42e48b83b...b3a11018ae1fe99190fb6fb7ae82a486c40f6f15 · ziglang/zig · GitHub)

nurpax · February 16, 2025, 7:42pm

Thanks! I believe the keychain change was merged on Feb 9 and the first failing nightly was updated to the docs on Feb 11.

Here’s my repro log:

➜  ~/dev/zig2d git:(main) ~/zig-macos-aarch64-0.14.0-dev.3066+1a1389c51/zig fetch --save git+https://github.com/natecraddock/ziglua
info: resolved to commit daa99a28a21523f680ebade522202440bbe720db
info: existing dependency named 'ziglua' is up-to-date

➜  ~/dev/zig2d git:(main) ~/zig-macos-aarch64-0.14.0-dev.3089+87bbb49d5/zig fetch --save git+https://github.com/natecraddock/ziglua
info: resolved to commit daa99a28a21523f680ebade522202440bbe720db
info: existing dependency named 'ziglua' is up-to-date

➜  ~/dev/zig2d git:(main) ~/zig-macos-aarch64-0.14.0-dev.3091+42e48b83b/zig fetch --save git+https://github.com/natecraddock/ziglua
info: resolved to commit daa99a28a21523f680ebade522202440bbe720db
info: existing dependency named 'ziglua' is up-to-date

➜  ~/dev/zig2d git:(main) ~/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/zig fetch --save git+https://github.com/natecraddock/ziglua
error: unable to discover remote git server capabilities: CertificateBundleLoadFailure

squeek502 · February 16, 2025, 7:47pm

Might be good to know where exactly it’s failing via the error return trace. Try running this with zig test:

const std = @import("std");

test "scan for OS-provided certificates" {
    var bundle: std.crypto.Certificate.Bundle = .{};
    defer bundle.deinit(std.testing.allocator);

    try bundle.rescan(std.testing.allocator);
}

nurpax · February 16, 2025, 7:52pm

Here’s the zig test output on build 3187:

➜  ~/dev/zig2d git:(main) ✗ ~/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/zig test fetch_repro.zig
1/1 fetch_repro.test.scan for OS-provided certificates...FAIL (CertificateFieldHasInvalidLength)
/Users/janne/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/lib/std/crypto/Certificate.zig:882:17: 0x100a58c57 in parse (test)
                return error.CertificateFieldHasInvalidLength;
                ^
/Users/janne/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/lib/std/crypto/Certificate.zig:397:29: 0x100a2403b in parse (test)
    const tbs_certificate = try der.Element.parse(cert_bytes, certificate.slice.start);
                            ^
/Users/janne/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/lib/std/crypto/Certificate/Bundle.zig:273:21: 0x100a0e313 in parseCert (test)
        else => |e| return e,
                    ^
/Users/janne/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/lib/std/crypto/Certificate/Bundle/macos.zig:74:17: 0x100a09b67 in rescanMac (test)
                try cb.parseCert(gpa, cert_start, now_sec);
                ^
/Users/janne/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/lib/std/crypto/Certificate/Bundle.zig:62:19: 0x100a04897 in rescan (test)
        .macos => return rescanMac(cb, gpa),
                  ^
/Users/janne/dev/zig2d/fetch_repro.zig:7:5: 0x100a047e3 in test.scan for OS-provided certificates (test)
    try bundle.rescan(std.testing.allocator);
    ^
0 passed; 0 skipped; 1 failed.
error: the following test command failed with exit code 1:
/Users/janne/dev/zig2d/.zig-cache/o/2f06da7503f669493dfd1d39105a034f/test --seed=0x9f0ae1b9

Also tried with build 3091 and the test passes.

squeek502 · February 16, 2025, 7:54pm

Could you try making this change to the failing zig install’s lib/std/crypto/Certificate/Bundle/macos.zig and then see if the test starts passing?

nurpax · February 16, 2025, 8:03pm

Seems to fail even with that change.

I updated it like this:

    // janne: Original
    // const keychainPaths = [2][]const u8{
    //     "/System/Library/Keychains/SystemRootCertificates.keychain",
    //     "/Library/Keychains/System.keychain",
    // };
    // Updated as per ziggit
    const keychainPaths = [2][]const u8{
        "/Library/Keychains/System.keychain",
        "/System/Library/Keychains/SystemRootCertificates.keychain",
    };

Here’s what I get (same as before, I think). I verified that the new code was being run by adding a couple of debug prints before the debug change.

➜  ~/dev/zig2d git:(main) ✗ ~/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/zig test fetch_repro.zig
1/1 fetch_repro.test.scan for OS-provided certificates...FAIL (CertificateFieldHasInvalidLength)
/Users/janne/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/lib/std/crypto/Certificate.zig:882:17: 0x100d74c57 in parse (test)
                return error.CertificateFieldHasInvalidLength;
                ^
/Users/janne/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/lib/std/crypto/Certificate.zig:397:29: 0x100d4003b in parse (test)
    const tbs_certificate = try der.Element.parse(cert_bytes, certificate.slice.start);
                            ^
/Users/janne/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/lib/std/crypto/Certificate/Bundle.zig:273:21: 0x100d2a313 in parseCert (test)
        else => |e| return e,
                    ^
/Users/janne/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/lib/std/crypto/Certificate/Bundle/macos.zig:80:17: 0x100d25b67 in rescanMac (test)
                try cb.parseCert(gpa, cert_start, now_sec);
                ^
/Users/janne/zig-macos-aarch64-0.14.0-dev.3187+d4c85079c/lib/std/crypto/Certificate/Bundle.zig:62:19: 0x100d20897 in rescan (test)
        .macos => return rescanMac(cb, gpa),
                  ^
/Users/janne/dev/zig2d/fetch_repro.zig:7:5: 0x100d207e3 in test.scan for OS-provided certificates (test)
    try bundle.rescan(std.testing.allocator);
    ^
0 passed; 0 skipped; 1 failed.
error: the following test command failed with exit code 1:
/Users/janne/dev/zig2d/.zig-cache/o/2f06da7503f669493dfd1d39105a034f/test --seed=0x68adf860

nurpax · February 16, 2025, 8:19pm

Modifying the debug change to not check for /Library/Keychains/System.keychain makes the test pass:

    const keychainPaths = [1][]const u8{
        // "/Library/Keychains/System.keychain",
        "/System/Library/Keychains/SystemRootCertificates.keychain",
    };

I guess there’s something unusual in my /Library/Keychains/System.keychain.

squeek502 · February 16, 2025, 8:23pm

Can you try making this change to line 72?

- cb.bytes.items.len += try reader.readAtLeast(dest_buf, cert_header.cert_size);
+ cb.bytes.items.len += try reader.readAll(dest_buf[0..cert_header.cert_size]);

nurpax · February 16, 2025, 8:28pm

The test seems to fail the same with this change.

squeek502 · February 16, 2025, 8:35pm

Good to know, then my theory of the bug is incorrect. Unfortunately I can’t reproduce the problem locally, so it’s hard to know what’s going wrong.

(the readAtLeast → readAll change probably makes sense regardless, though, as it seems like the use of readAtLeast causes cb.bytes to balloon by millions of bytes [last cert starting at index 173295 with readAll vs 27485590 with readAtLeast, final capacity of 251937 with readAll vs 32720747 with readAtLeast])

squeek502 · February 18, 2025, 5:54am

For anyone curious, @nurpax and I did some sleuthing and this is the fix that was found to work:

github.com/ziglang/zig

Skip empty/invalid records/certs in MacOS keychain files

ziglang:master ← squeek502:macos-invalid-keychain-skip

opened 12:24AM - 18 Feb 25 UTC

squeek502

+6 -0

In the original PR that implemented `rescanMac` (https://github.com/ziglang/zig/…pull/14325), it included a list of references for the keychain format. Multiple of those references include the checks that are added in this commit, and empirically this fixes the loading of a real keychain file that was previously failing (it had both a record with offset 0 and a record with cert_size 0). To give more detail, when not skipping a record with offset zero, it would attempt to read the base of the table as a `X509CertHeader`, but the base of the table is actually a `TableHeader`. It would then take this misinterpreted `X509CertHeader` and read N bytes that follow it (where N is equal to the misinterpreted `cert_size`), and then try to parse those bytes as a DER cert. These bytes are just some arbitrary bytes in the keychain file, though, not the actual start of a cert, so it would fail at some arbitrary point in `Certificate.parse`. Fixes #22870 --- Specifically, this has fixed the regression for @nurpax detailed in https://ziggit.dev/t/zig-fetch-failing-with-discover-remote-git-server-capabilities-certificatebundleloadfailure/8608 However, this fix does not explain/account for the 'swap the ordering' workaround from https://github.com/ziglang/zig/issues/22870#issuecomment-2653836145, so this may not be a full solution. Would appreciate @jedisct1 checking to see that this fixes the regression for them.

(also submitted the readAtLeast change mentioned above here)