Zig Build System Basics

33 Likes

An interesting comment on Youtube by @Surkimus

FYI, if you start your chapter list in the description with a 0:00, it will be also available in the video scrubber

thanks, fixed!

I feel like the important fact that ā€œThe build system is a DSL for creating a build graphā€ could really come across a bit better in the build system API - so that it is harder to do the wrong thing without depending too much on code examples and documentation.

E.g. the build graph construction looks to imperative, and has quite confusing workarounds for data items that are resolved at build time (lazyPath, ā€˜resolved target’, …), currently it’s also impossible to run arbitrary Zig code at build time except by moving that code into a separate executable. There must be a better way to describe such a two-stage process (first describe the build graph via Zig, but also inject regular Zig functions into the graph to run at build time). Hopefully once all the requirements for the Zig build system have somewhat stabilized, there can be an API overhaul from the ground up designed for those requirement. Just my 2c :slight_smile:

7 Likes

Running arbitrary code at build time is overkill for many things and potentially tricky to audit for security. I think the current way is good in that allows you to run arbitrary Zig code but makes you work a bit for it.

From a security perspective, is there a relevant difference between the risk of build.zig containing malicious code or e.g. a test (zig build test) containing malicious code?

2 Likes

At least tests are opt-in, but really there’s no way around auditing code you don’t trust.

Even running zig build --help is enough if build.zig contains malicious code.

You might find this sandbox idea interesting: Run build.zig logic in a WebAssembly sandbox Ā· Issue #14286 Ā· ziglang/zig Ā· GitHub

3 Likes

You can run arbitrary code in the build.zig, so yeah if you don’t trust the author, don’t run their code, whether that’s a build script or not. Also note that this applies to most other build systems, like e.g. CMake, as well, because they allow running arbitrary batch/bash files.

Sounds interesting and something I’m willing to do, but admittedly I’m not sure exactly what changes would be made. Perhaps when the time comes you’d be willing to submit a sketch as a patch to explain the alternate API ideas?

4 Likes

I agree, which is why I’ve been wanting some sort of build graph debugging tool for my larger projects, like a generated SVG image of the graph, for example. Visual Build Graph

I’ll see if I can contribute some useful feedback when the time comes :wink: I’ve been running into similar problems with a cmake wrapper project I’ve been tinkering with (with the difference that there’s 3 phases: (1) describing the cmake structure in Typescript, (2) the cmake generation phase, (3) the build execution phase…

In general, for the build declaration phase I think it might help to reduce the ā€˜API surface’ to consider for the happy/simple path… e.g. fewer individual function calls, but maybe allow to describe the build graph with a single call which describes the entire tree with a nested stuct, and only fall back to function calls for special cases (like creating build steps in a loop, or complicated if/else logic).

Ideally of course it would be possible to mix the ā€˜build tree as a struct’ with ā€˜injected’ Zig code which implements more complicated logic right inside the struct initialization (which should be possible via init: { } blocks…

Also maybe it would help a lot with communicating the idea that the build API is declarative by having an actual BuildGraph object which must be returned from the ā€˜main’ build function (instead of ā€œappendingā€ to the Build object).

…but yeah, I’ll try to come up with something more coherent at a later point…

My only real problem with the build system atm is that it’s very hard to find information about it. The docs are good but too concise and don’t have enough examples. Videos like these are very helpful but a long list of recipes would be nice to have too.

1 Like

We can always add more to it.

6 Likes

I agree the build system is very interesting and powerful,
With it I can fetch zip, extract them, parch them, include C headers in my zig project, link the binary, run command to find system library for platform like IOS or android, add in the build directory every file I want and of course build zig.

It’s really a powerful and versatile tooling however with the recent breaking changes finding example for all of this has become harder. I did find with other GitHub repo every I needed but the tiny example of the official learning source don’t explained anything on how to add the android sysroot for compiling internal C libraries like miniaudio where they need Ā« pthread Ā» to works.

As always, zig is powerful, got a lot of potential, a great community but lack of a good documentation on the build system.

2 Likes

Of course, and we should! But I don’t understand the internals so as a user who wants to get the build system Do The Thing :trade_mark: the only thing holding me back is that it’s under-documented, which is a good problem to have since it’s one more easily fixed than many others.

Edit: That page didn’t show up when I searched for formatting generated zig code, which is a topic that’s in there. I’ll make sure to remember it

1 Like

just for clarity, I don’t think lazy paths and resolved targets should be put in the same category. Lazy paths are indeed a way of referring to paths that will be solved at make time (that’s the official name of the second phase of the build script where the steps are actually executed), but a resolved target is just a way of filling in the gaps when the target triple you specify doesn’t have all the details in it, for example when you request the native target. The target resolution happens in the configure phase alongside the rest of your build script.

Given how the build system (and the Zig language) works as a whole, the build system could provide some sugar to make the process less tedious, but at the end of the day you have to take user code, turn it into an executable and then run it in the make phase, at the very least so that you can correctly integrate with how caching works. The ā€œsugarā€ provided by the build system could let you just provide a path to a zig script with a main function to be compiled and run as a a step and that would be probably nice to have, but it would still be something that works up to a point as the instant you start needing dependencies, then you would be back to defining an executable like you already have to do now.

Back when I just started learning Zig, I didn’t bother learning the build system for a relatively long time (6mo, maybe a year?) but then once I bit the bullet the whole system turned out to be fairly easy to understand, especially after lazy paths were introduced. Not to say that the build system is perfect (I’m sure that every single API of the build system could be improved by a non-trivial margin), but I fear a more functional approach to graph building would end up creating a system that is easier to get started with, but that ends up becoming not as easy as the current one once you become an advanced user.

4 Likes

Yeah that’s the tricky part ā€œsimple things should be simple, complex things should be possibleā€ etc…

The only potential problem I see long-term is that experienced users might develop a ā€˜blindness’ towards problems in the build system API design…

Generally I think that the changes in the build system API have been going into the right direction, e.g. I don’t remember if the .imports field was always there when declaring a module, but I think this:

    const mod_chipz = b.addModule("chipz", .{
        // ...
        .imports = &.{
            .{ .name = "common", .module = mod_common },
            .{ .name = "chips", .module = mod_chips },
            .{ .name = "systems", .module = mod_systems },
            .{ .name = "host", .module = mod_host },
        },
    });

…is much better than this:

    const mod_chipz = b.addModule("chipz", .{
        // ...
    });
    mod_chipz.addImport("common", mod_common);
    mod_chipz.addImport("chips", mod_chips);
    mod_chipz.addImport("systems", mod_systems);
    mod_chipz.addImport("host", mod_host);

…and this example is essentially the gist of where I’d like to see the build system API moving. The addImport call might still be needed for flexibility (e.g. adding imports in a loop, or conditionally add imports), but IMHO it might be nicer to only have the struct approach if it is just as convenient to initialize a struct via ā€˜logic’ (the obvious solution would be init: { } blocks, but maybe there are better ways.

PS: now - whether this means that describing the entire build graph as a single big nested struct is a good idea, I don’t know, and I have my doubts too tbh… I do like the idea of the build() function returning a BuildGraph object (no matter how this BuildGraph object is actually created), it might help to bring the idea across that the code that runs in build() is purely declarative, e.g. ā€œthe job of the build() function is to describe the build by creating and returning a single BuildGraph objectā€ā€¦

…and - just going wild here - maybe integrating external dependencies into the build means that I’m getting the BuildGraph object of the dependency which I then ā€˜hook’ into the top level BuildGraph? (and maybe I can even inspect and manipulate the dependency BuildGraph object in the toplevel build.zig). Anyway, over and out :slight_smile:

3 Likes

are you sure it’s not best to leave it imperative as a low level ā€œprimitivesā€ to produce a build graph, and instead exploring new APIs at library-level? comptime is top for creation of declarative configuration-like APIs and other groupings like you suggest.

I feel like when sandboxing lands, it’s less of a problem since IO functions would be a compile error.

Maybe, but then, where does such a build system wrapper library live? The best place would be the stdlib to avoid ecosystem fragmentation, but then there would be two build system APIs in the stdlib (a high level and a low level API - not sure if that’s a good idea).

In a way I’m already doing things like this by exposing build-step creation helpers from the sokol-zig build.zig file (e.g. a helper function which returns a shader compiler build step, and a helper function to return an Emscripten linker build step - both are just wrappers around addSystemCommand). But this means that the sokol build.zig needs to be imported as module - which brings me back to the topic of allowing to define custom build steps that can be ā€˜registered’ with the build system without having to import a dependency’s build.zig as a module :wink:

1 Like