lazyDependency, fetch guard and zig-pkg cache surprising behavior

Help
1

Consider the following build script stub :

pub fn build(b: *std.Build) void {
    const opt_fetch = b.option(
        bool,
        "fetch",
        "this should prevent any fetch of the package",
    ) orelse false;
    if (opt_fetch) {
        if (b.lazyDependency("sdl", .{})) |sd| {
            const step_install_sdl_tree = b.addInstallDirectory(
                .{
                    .source_dir = sd.artifact("SDL2").getEmittedIncludeTree(),
                    .install_dir = .prefix,
                    .install_subdir = "sdl2_tree",
                },
            );
            _ = step_install_sdl_tree; // autofix
        }
    }
}

SDL dependency is not really relevant, just happens to trigger an error when fetched on zig 0.17 because it uses b.sysroot that was removed.

build.zig.zon file I'm testing it with .{ .name = .lazy_fetch_behavior, .version = "0.0.1", .minimum_zig_version = "0.17.0-dev.813+2153f8143", .paths = .{""}, .dependencies = .{ .sdl = .{ .url = "git+https://github.com/david-vanderson/SDL#b25735a465f7db1c160d2eca8ac46dde9ec41cfb", .hash = "SDL-2.32.10-JToi37G3EgG0eraPos4vhCwC6vVgyB1FYf1SxxhpBr_Y", .lazy = true, }, }, .fingerprint = 0x6fabe45dbfdfd5be, }

If I run a bare zig build nothing happens after the configuration phase as expected.

I’m not entirely sure why the opt_fetch guard is even required, my naive assumption would be that the fetch only occurs if a step actually depends on the lazyDependency but maybe there is a good technical reason for that.

Then let’s say you try the dependency (i.e. zig build -Dfetch=true), it download, recompress, and in that particular case it triggers a build error related to b.sysroot missing.

Ok, so, you go back to zig build -Dfetch=false and the error is still there.
After being confused for a while, I realize it was already fetched, so I rm -r zig-pkg but the error persists.
It took me a while to realize that I had to rm -r ~/.cache/zig/p/SDL-... to “revert” to a state where I don’t trigger the error.

I very much like the design architecture with zig-pkg + recompressed global cache, but I have to say that the semantics of when which is used is not super transparent.
And everything is so fast that it’s tricky to follow :sweat: (Is it a download ? Is it a cache hit ? Don’t know, don’t care, it’s just zig and it’s fast. ahahahah. Ah, no more disc space third time today ? Right, I have 500 .zig-cache folders in my trash bin again. :partying_face: I’m having a lot of fun.)

Anyways, shouldn’t the analysis of the package be skipped in that case regardless of the presence/absence of package in cache ? Am I missing something ?

2

There are multiple factors at play here.

The first is that there’s an important distinction between comptime-known and runtime-known conditions. Imagine you have something like the following:

if (x) {
    std.debug.print("oh yes!\n", .{});
} else {
    @compileError("oh no!");
}

If x is comptime-known to be true, the code will compile because the @compileError is never analyzed. However, if x is runtime-known (for example, a non-comptime function parameter), both branches will be analyzed and the code will fail to compile, even if in practice all code paths always ensure x is true at runtime. So whether or not opt_fetch is true or false is irrelevant, because the compiler will always the branch either way and if the branch contains code that doesn’t compile, compilation will fail.

Second, the way the build system works behind the scenes is that it generates a file named dependencies.zig which contains information about and the imported build.zigs for all dependencies that have been fetched:

pub const packages = struct {
    pub const @"aro-0.0.0-JSD1Qum6OgB83hmRWguTl2Z9x3rEjHWIu8f_RsMXJTTg" = struct {
        pub const build_root = "C:\\temp\\zig-examples\\breakout\\zig-pkg\\aro-0.0.0-JSD1Qum6OgB83hmRWguTl2Z9x3rEjHWIu8f_RsMXJTTg";
        pub const build_zig = @import("aro-0.0.0-JSD1Qum6OgB83hmRWguTl2Z9x3rEjHWIu8f_RsMXJTTg");
        pub const deps: []const struct { []const u8, []const u8 } = &.{
        };
    };
    pub const @"sdl-0.5.1+3.4.10-SDL--kbMpgGMXke11Ujh5HUPKch7G_SUAS12LI0QFoqj" = struct {
        pub const build_root = "C:\\temp\\zig-examples\\breakout\\zig-pkg\\sdl-0.5.1+3.4.10-SDL--kbMpgGMXke11Ujh5HUPKch7G_SUAS12LI0QFoqj";
        pub const build_zig = @import("sdl-0.5.1+3.4.10-SDL--kbMpgGMXke11Ujh5HUPKch7G_SUAS12LI0QFoqj");
        pub const deps: []const struct { []const u8, []const u8 } = &.{
            .{ "sdl_linux_deps", "sdl_linux_deps-0.0.0-SDL_ltg8hgAOayMwFN6BhHW3Rs5UPtf6n8m-2qcQHuGS" },
        };
    };
    pub const @"sdl_linux_deps-0.0.0-SDL_ltg8hgAOayMwFN6BhHW3Rs5UPtf6n8m-2qcQHuGS" = struct {
        pub const available = true;
        pub const build_root = "C:\\temp\\zig-examples\\breakout\\zig-pkg\\sdl_linux_deps-0.0.0-SDL_ltg8hgAOayMwFN6BhHW3Rs5UPtf6n8m-2qcQHuGS";
        pub const build_zig = @import("sdl_linux_deps-0.0.0-SDL_ltg8hgAOayMwFN6BhHW3Rs5UPtf6n8m-2qcQHuGS");
        pub const deps: []const struct { []const u8, []const u8 } = &.{
        };
    };
    pub const @"translate_c-0.0.0-Q_BUWuEVBwCbZUsNorTQ6-d3nYx8KDgHHWtVWBzgzyB5" = struct {
        pub const build_root = "C:\\temp\\zig-examples\\breakout\\zig-pkg\\translate_c-0.0.0-Q_BUWuEVBwCbZUsNorTQ6-d3nYx8KDgHHWtVWBzgzyB5";
        pub const build_zig = @import("translate_c-0.0.0-Q_BUWuEVBwCbZUsNorTQ6-d3nYx8KDgHHWtVWBzgzyB5");
        pub const deps: []const struct { []const u8, []const u8 } = &.{
            .{ "aro", "aro-0.0.0-JSD1Qum6OgB83hmRWguTl2Z9x3rEjHWIu8f_RsMXJTTg" },
        };
    };
};

pub const root_deps: []const struct { []const u8, []const u8 } = &.{
    .{ "sdl", "sdl-0.5.1+3.4.10-SDL--kbMpgGMXke11Ujh5HUPKch7G_SUAS12LI0QFoqj" },
    .{ "translate_c", "translate_c-0.0.0-Q_BUWuEVBwCbZUsNorTQ6-d3nYx8KDgHHWtVWBzgzyB5" },
};

This generated file is exposed to std.Build as build_runner.dependencies. b.dependency and all of its variants reference this struct:

pub fn dependency(b: *Build, name: []const u8, args: anytype) *Dependency {
    const build_runner = @import("root");
    const deps = build_runner.dependencies;
    const pkg_hash = findPkgHashOrFatal(b, name);

    inline for (@typeInfo(deps.packages).@"struct".decl_names) |decl_name| {
        if (mem.eql(u8, decl_name, pkg_hash)) {
            const pkg = @field(deps.packages, decl_name);
            if (@hasDecl(pkg, "available")) {
                panic("dependency '{s}{s}' is marked as lazy in build.zig.zon which means it must use the lazyDependency function instead", .{ b.dep_prefix, name });
            }
            return dependencyInner(b, name, pkg.build_root, if (@hasDecl(pkg, "build_zig")) pkg.build_zig else null, pkg_hash, pkg.deps, args);
        }
    }

    unreachable; // Bad @dependencies source
}

An important detail here is that because the name parameter is runtime-known, all build_zig decls and all build functions will be analyzed by the compiler, so if any of them contain invalid code, it is a compile error even if your particular zig build invocation won’t reach the b.dependency() call.

Third, there’s also lazy dependencies and how they interact with the build. I’ve explained how lazyDependency works and why the guard is necessary in the past so I’ll link to that post instead of repeating myself here. However, the key detail here is that once a lazy dependency has been fetched, the generated dependencies.zig file is regenerated to include it, meaning that if a lazy dependency has a broken build.zig file, your build will compile and work fine until the dependency is fetched, after which you will start getting compile errors.

However, I would argue that it is unexpected and arguably a bug that Zig eagerly fetches and unpacks the lazy dependency to zig-pkg if it happens to be present in the global cache. Deleting zig-pkg and .zig-cache should be enough to fix your build (and even ignoring this specific “dependency build.zig doesn’t compile” case, eagerly unpacking lazy dependencies is wastes disk space for no reason and is undesirable in other cases as well).

Consider opening an issue for this if no one has done so already.

2 Likes