A custom build.zig command to run system commands and see their output

Hello.

I have been banging my head at this for the better part of three hours. And I cannot figure out how this is solved.

As a background, I am building a toy C compiler in Zig based on Nora Sandler’s Writing a C Compiler book. And I am trying to make some testing tasks go through build.zig rather than relying on shell scripts.

The project itself (and build.zig) lives in a directory different from. The test suite lives in another directory that I want to pass to the command. What I am working towards is calling the command this way zig build eye -- path/to/tests/ --flag and it would traverse the folder calling bat on each file, then calling my executable (with --flag) so I can visually compare on the terminal the C file and its output.

This is the current build.zig:

const std = @import("std");
pub fn build(b: *std.Build) !void {
    const exe_mod = b.createModule(.{
        .root_source_file = b.path("src/main.zig"),

        // this is the target for the Book and my machine.
        .target = b.resolveTargetQuery(.{
            .cpu_arch = .x86_64,
            .os_tag = .macos,
        }),
        .optimize = switch (b.release_mode) {
            .off => .Debug,
            .safe => .ReleaseSafe,
            else => .ReleaseFast,
        },
    });

    const fmt_step = b.addFmt(.{ .paths = &.{"./src/"} });

    const exe = b.addExecutable(.{
        .name = "paella",
        .root_module = exe_mod,
    });
    exe.step.dependOn(&fmt_step.step);

    b.installArtifact(exe);

    { // `zig build run` command
        const run_step = b.step("run", "Run the app");

        const run_cmd = b.addRunArtifact(exe);
        if (b.args) |args| // pass aruments into `zig build run --`
            run_cmd.addArgs(args);

        run_cmd.step.dependOn(b.getInstallStep());
        run_step.dependOn(&run_cmd.step);
    }
    // two steps omitted here because they work fine.
    { // `zig build eye` command
        const eye_step = b.step("eye", "eye test all the files in a given directory");

        var closure = b.allocator.create(Closure) catch unreachable;
        closure.* = .{ .exe = exe, .step = std.Build.Step.init(.{
            .id = .custom,
            .name = "inner_eye",
            .makeFn = make_eye_step,
            .owner = b,
        }) };

        eye_step.dependOn(&closure.step);
    }
}

const Closure = struct {
    exe: *std.Build.Step.Compile,
    step: std.Build.Step,
};

fn make_eye_step(step: *std.Build.Step, _: std.Build.Step.MakeOptions) !void {
    const b = step.owner;
    const closure: *const Closure = @fieldParentPtr("step", step);
    const exe = closure.exe;

    const args: []const []const u8 = b.args orelse
        &.{ "./c_files/", "--lex" }; // least harmful default;

    const lazy = b.path(args[0]);
    const path = lazy.getPath3(b, null);

    const dir = try path.openDir("", .{ .access_sub_paths = false });
    var walker = try dir.walk(b.allocator);
    defer walker.deinit();

    var prev_run_cmd: ?*std.Build.Step.Run = null;

    while (try walker.next()) |entry| {
        if (entry.kind == .file and std.mem.endsWith(u8, entry.basename, ".c")) {
            const file = entry.path;

            const bat = b.addSystemCommand(&.{ "bat", file });
            bat.setCwd(lazy);
            bat.stdio = .inherit;

            if (prev_run_cmd) |c|
                bat.step.dependOn(&c.step);

            const run_cmd = b.addRunArtifact(exe);
            run_cmd.setCwd(lazy);
            run_cmd.addArg(file);
            run_cmd.addArgs(args[1..]);
            run_cmd.stdio = .inherit;

            run_cmd.step.dependOn(b.getInstallStep());

            run_cmd.step.dependOn(&bat.step);
            prev_run_cmd = run_cmd;
        }
    }


    step.dependOn(&prev_run_cmd.?.step);
}

When the loop is in the main function, it runs every time I run zig build which is … unwanted behaviour. Someone on the Zig discord suggested putting it on a separate function and calling a custom step like that. But now while everything compiles , the commands don’t seem to even run.

What am I missing? I can’t make head or tails of anything in the standard library or the online documentation.

I can just as well create a shell script and run it from build.zig, but I want to make this work within Zig itself.

You can simplify your dependency tree a little bit. Have the eye step depend on the install step rather than having each run_cmd step depend on it. I also don’t see a need for the inner_eye step… I would build the dependency graph based on the eye step, give the make_eye_step whatever signature you want and get rid of the makeFn and @fieldParentPtr stuff.

I am not sure I understand what you suggesting .

You can simplify your dependency tree a little bit.

I want to understand why it is not working. Everything seems to depend on the correct things.

Have the eye step depend on the install step

This is the same as the default setup for zig build run which works fine. Why isn’t it working here?

I also don’t see a need for the inner_eye step… I would build the dependency graph based on the eye step,

The problem that if I put the loop in the main function it would run every time I ran zig build when I want it to only run for that specific step. This is the solution that was suggested to me and I also found here

give the make_eye_step whatever signature you want and get rid of the makeFn and @fieldParentPtr stuff.

I have absolutely no idea what this means. How can I make sure otherwise that the directory walk does not loop for every other zig build command ?

Hey, welcome to Ziggit!

As I understand it the build process is split into two main stages, the ‘configure’ phase and the ‘make’ phase.
During the configure phase, the build graph is constructed. During the make phase, it is executed. These two phases are supposed to be completely separate and mixing them might result in unwanted behaviour. However as of now there aren’t really any safeguards to check this nor is there any documentation about which function exactly is allowed when (except for the occasional code comment).

Relevant issue: introduce build_runner.phase enum {configure, make} and assert the phase in many functions · Issue #14941 · ziglang/zig · GitHub

You are conflating the two stages by trying to edit your build graph while it is already being executed by creating steps inside a make function which I’m pretty sure is not allowed. That’s probably why the build script doesn’t work the way you expect.

To fix this you would probably have to have the loop walking the dir and creating build steps in your main build function anyways and then make all the steps you create during the walk depend on some other step that only gets executed if you invoke zig build eye.

1 Like

Thank you. The explanation makes perfect sense.

That leaves the problem of how I make it the directory walk does not run when I am invoking another build command than this one? Because I am not always running the same arguments or in the same order.

You need the directory walk to build your dependency tree, and you can still create the full tree when you run other commands. The difference is when you do not specify the eye step the build system will see that as a branch that does not need to be executed. Something like this is what I was suggesting:

const std = @import("std");
pub fn build(b: *std.Build) !void {
    const exe_mod = b.createModule(.{
        .root_source_file = b.path("src/main.zig"),

        // this is the target for the Book and my machine.
        .target = b.resolveTargetQuery(.{
            .cpu_arch = .x86_64,
            .os_tag = .macos,
        }),
        .optimize = switch (b.release_mode) {
            .off => .Debug,
            .safe => .ReleaseSafe,
            else => .ReleaseFast,
        },
    });

    const fmt_step = b.addFmt(.{ .paths = &.{"./src/"} });

    const exe = b.addExecutable(.{
        .name = "paella",
        .root_module = exe_mod,
    });
    exe.step.dependOn(&fmt_step.step);

    b.installArtifact(exe);

    { // `zig build run` command
        const run_step = b.step("run", "Run the app");

        const run_cmd = b.addRunArtifact(exe);
        if (b.args) |args| // pass aruments into `zig build run --`
            run_cmd.addArgs(args);

        run_cmd.step.dependOn(b.getInstallStep());
        run_step.dependOn(&run_cmd.step);
    }
    // two steps omitted here because they work fine.
    { // `zig build eye` command
        const eye_step = b.step("eye", "eye test all the files in a given directory");
        make_eye_step(b, exe, eye_step);
        eye_step.dependOn(b.getInstallStep());
    }
}

fn make_eye_step(b: *std.Build, exe: *std.Build.Step.Compile, eye_step: *std.Build.Step) !void {
    const args: []const []const u8 = b.args orelse
        &.{ "./c_files/", "--lex" }; // least harmful default;

    const lazy = b.path(args[0]);
    const path = lazy.getPath3(b, null);

    const dir = try path.openDir("", .{ .access_sub_paths = false });
    var walker = try dir.walk(b.allocator);
    defer walker.deinit();

    var prev_run_cmd: ?*std.Build.Step.Run = null;

    while (try walker.next()) |entry| {
        if (entry.kind == .file and std.mem.endsWith(u8, entry.basename, ".c")) {
            const file = entry.path;

            const bat = b.addSystemCommand(&.{ "bat", file });
            bat.setCwd(lazy);
            bat.stdio = .inherit;

            if (prev_run_cmd) |c|
                bat.step.dependOn(&c.step);

            const run_cmd = b.addRunArtifact(exe);
            run_cmd.setCwd(lazy);
            run_cmd.addArg(file);
            run_cmd.addArgs(args[1..]);
            run_cmd.stdio = .inherit;
            run_cmd.step.dependOn(&bat.step);
            prev_run_cmd = run_cmd;
        }
    }

    eye_step.dependOn(&prev_run_cmd.?.step);
}

Thank you a lot. I was hacking around on it since my last reply and came up with a solution that works (almost) the way I originally imagined, and I think is more in tune with the system.

  { // `zig build eye` command
        const eye_step = b.step("eye", "eye test all the files in a given directory");

        if (b.option(std.Build.LazyPath, "folder", "Path to eye")) |lazy| {
            const path = lazy.getPath3(b, null);

            const dir = try path.openDir("", .{ .access_sub_paths = false });
            var walker = try dir.walk(b.allocator);
            defer walker.deinit();

            var prev_run_cmd: ?*std.Build.Step.Run = null;

            while (try walker.next()) |entry| if (entry.kind == .file and
                std.mem.endsWith(u8, entry.basename, ".c"))
            {
                const file = entry.path;

                const bat = b.addSystemCommand(&.{ "bat", file });
                bat.setCwd(lazy);
                bat.stdio = .inherit;

                if (prev_run_cmd) |c|
                    bat.step.dependOn(&c.step);

                const run_cmd = b.addRunArtifact(exe);
                run_cmd.setCwd(lazy);
                run_cmd.addArg(file);
                run_cmd.addArgs(b.args orelse &.{});
                run_cmd.stdio = .inherit;

                run_cmd.step.dependOn(b.getInstallStep());
                run_cmd.step.dependOn(&bat.step);

                prev_run_cmd = run_cmd;
            };

            eye_step.dependOn(&prev_run_cmd.?.step);
        } else {
            // magic here
            const fail = b.addFail("folder needed for eye");
            eye_step.dependOn(&fail.step);
        }
    }

the only difference between this and how I originally envisioned it is is I am passing the folder before the -- rather than after. Either way, this does not run the tree walk in other commands and doesn’t fail in other commands when I omit this option. So … great all around

Thank you @alp and @Justus2308 . Your comments have been of great help.

2 Likes