lazyDependency, fetch guard and zig-pkg cache surprising behavior

Help
2

There are multiple factors at play here.

The first is that there’s an important distinction between comptime-known and runtime-known conditions. Imagine you have something like the following:

if (x) {
    std.debug.print("oh yes!\n", .{});
} else {
    @compileError("oh no!");
}

If x is comptime-known to be true, the code will compile because the @compileError is never analyzed. However, if x is runtime-known (for example, a non-comptime function parameter), both branches will be analyzed and the code will fail to compile, even if in practice all code paths always ensure x is true at runtime. So whether or not opt_fetch is true or false is irrelevant, because the compiler will always the branch either way and if the branch contains code that doesn’t compile, compilation will fail.

Second, the way the build system works behind the scenes is that it generates a file named dependencies.zig which contains information about and the imported build.zigs for all dependencies that have been fetched:

pub const packages = struct {
    pub const @"aro-0.0.0-JSD1Qum6OgB83hmRWguTl2Z9x3rEjHWIu8f_RsMXJTTg" = struct {
        pub const build_root = "C:\\temp\\zig-examples\\breakout\\zig-pkg\\aro-0.0.0-JSD1Qum6OgB83hmRWguTl2Z9x3rEjHWIu8f_RsMXJTTg";
        pub const build_zig = @import("aro-0.0.0-JSD1Qum6OgB83hmRWguTl2Z9x3rEjHWIu8f_RsMXJTTg");
        pub const deps: []const struct { []const u8, []const u8 } = &.{
        };
    };
    pub const @"sdl-0.5.1+3.4.10-SDL--kbMpgGMXke11Ujh5HUPKch7G_SUAS12LI0QFoqj" = struct {
        pub const build_root = "C:\\temp\\zig-examples\\breakout\\zig-pkg\\sdl-0.5.1+3.4.10-SDL--kbMpgGMXke11Ujh5HUPKch7G_SUAS12LI0QFoqj";
        pub const build_zig = @import("sdl-0.5.1+3.4.10-SDL--kbMpgGMXke11Ujh5HUPKch7G_SUAS12LI0QFoqj");
        pub const deps: []const struct { []const u8, []const u8 } = &.{
            .{ "sdl_linux_deps", "sdl_linux_deps-0.0.0-SDL_ltg8hgAOayMwFN6BhHW3Rs5UPtf6n8m-2qcQHuGS" },
        };
    };
    pub const @"sdl_linux_deps-0.0.0-SDL_ltg8hgAOayMwFN6BhHW3Rs5UPtf6n8m-2qcQHuGS" = struct {
        pub const available = true;
        pub const build_root = "C:\\temp\\zig-examples\\breakout\\zig-pkg\\sdl_linux_deps-0.0.0-SDL_ltg8hgAOayMwFN6BhHW3Rs5UPtf6n8m-2qcQHuGS";
        pub const build_zig = @import("sdl_linux_deps-0.0.0-SDL_ltg8hgAOayMwFN6BhHW3Rs5UPtf6n8m-2qcQHuGS");
        pub const deps: []const struct { []const u8, []const u8 } = &.{
        };
    };
    pub const @"translate_c-0.0.0-Q_BUWuEVBwCbZUsNorTQ6-d3nYx8KDgHHWtVWBzgzyB5" = struct {
        pub const build_root = "C:\\temp\\zig-examples\\breakout\\zig-pkg\\translate_c-0.0.0-Q_BUWuEVBwCbZUsNorTQ6-d3nYx8KDgHHWtVWBzgzyB5";
        pub const build_zig = @import("translate_c-0.0.0-Q_BUWuEVBwCbZUsNorTQ6-d3nYx8KDgHHWtVWBzgzyB5");
        pub const deps: []const struct { []const u8, []const u8 } = &.{
            .{ "aro", "aro-0.0.0-JSD1Qum6OgB83hmRWguTl2Z9x3rEjHWIu8f_RsMXJTTg" },
        };
    };
};

pub const root_deps: []const struct { []const u8, []const u8 } = &.{
    .{ "sdl", "sdl-0.5.1+3.4.10-SDL--kbMpgGMXke11Ujh5HUPKch7G_SUAS12LI0QFoqj" },
    .{ "translate_c", "translate_c-0.0.0-Q_BUWuEVBwCbZUsNorTQ6-d3nYx8KDgHHWtVWBzgzyB5" },
};

This generated file is exposed to std.Build as build_runner.dependencies. b.dependency and all of its variants reference this struct:

pub fn dependency(b: *Build, name: []const u8, args: anytype) *Dependency {
    const build_runner = @import("root");
    const deps = build_runner.dependencies;
    const pkg_hash = findPkgHashOrFatal(b, name);

    inline for (@typeInfo(deps.packages).@"struct".decl_names) |decl_name| {
        if (mem.eql(u8, decl_name, pkg_hash)) {
            const pkg = @field(deps.packages, decl_name);
            if (@hasDecl(pkg, "available")) {
                panic("dependency '{s}{s}' is marked as lazy in build.zig.zon which means it must use the lazyDependency function instead", .{ b.dep_prefix, name });
            }
            return dependencyInner(b, name, pkg.build_root, if (@hasDecl(pkg, "build_zig")) pkg.build_zig else null, pkg_hash, pkg.deps, args);
        }
    }

    unreachable; // Bad @dependencies source
}

An important detail here is that because the name parameter is runtime-known, all build_zig decls and all build functions will be analyzed by the compiler, so if any of them contain invalid code, it is a compile error even if your particular zig build invocation won’t reach the b.dependency() call.

Third, there’s also lazy dependencies and how they interact with the build. I’ve explained how lazyDependency works and why the guard is necessary in the past so I’ll link to that post instead of repeating myself here. However, the key detail here is that once a lazy dependency has been fetched, the generated dependencies.zig file is regenerated to include it, meaning that if a lazy dependency has a broken build.zig file, your build will compile and work fine until the dependency is fetched, after which you will start getting compile errors.

However, I would argue that it is unexpected and arguably a bug that Zig eagerly fetches and unpacks the lazy dependency to zig-pkg if it happens to be present in the global cache. Deleting zig-pkg and .zig-cache should be enough to fix your build (and even ignoring this specific “dependency build.zig doesn’t compile” case, eagerly unpacking lazy dependencies is wastes disk space for no reason and is undesirable in other cases as well).

Consider opening an issue for this if no one has done so already.

2 Likes