How to call other top-level steps with arguments?

Help
1

Let’s say I have a Zig project that uses build.zig which provides various steps, like zig build test. The steps take various config options like:

zig build test -Drelease
zig build example -Dexample-name="hello world"

Now, let’s say I have my CI script, which invokes a certain sequence of steps:

- zig buid test
- zig build test -Drelease
- zig build example -Dexample-name="hello world"
- zig build example -Dexample-name="hello world" -Dtarget=x86_64-windwos

That is horrible yaml with is painful to work with. Ideally, I’d love to reduce my ci script to just

- zig build ci

and have that ci step call appropriate steps with various arguments. What’s the best way to do that? The problem here is that, in build.zig, only one set o arguments exists at the same time.

I can call addSystemCommand myself a bunch, which I think would work:

    const step_ci = b.step("ci", "Run full suite of checks");
    step_ci.dependOn(
        &b.addSystemCommand(.{ b.graph.zig_exe, "build", "test", "-Drelease" }).step,
    );
    step_ci.dependOn(
        &b.addSystemCommand(.{ b.graph.zig_exe, "build", "test" }).step,
    );

but one problem here is that I don’t get a unified build graph, so this won’t necessary use my CPU cores most efficiently and will have suboptimal progress reporting.

Is there a better pattern here?

2

I’m sure we’re all aware that most CI platforms support matrix strategies:

But it’s obviously more maintainable to handroll a ci step in build.zig. However, it’ll need some forethought.

I might be missing something, but this should work for the “Run full suite of checks” CI use case:

  • Define static matrices of values as:
    • Enum:
    const Example = enum {
    example1,
    example2,
};
    • Tuple of strings:
    const RELEASE_TRIPLES = .{
    "aarch64-macos",
    "x86_64-linux",
    "x86_64-windows",
};
  • inline for loop through those values and call ci_step.dependOn(&<artifact>.step).

Just as a reference for such loops, see steps from liza’s templates:

3 Likes
3

The problem with that is that you have to duplicate step definitions for:

  • build one step based on passed -D flags
  • build all the steps based on the matrix.

Eg, in your example, these two are essentially the same definition:

This is not optimal, for two reasons:

  • If the step definition is complex, you get a lot of copy-paste. Which you can battle with extract step-creating function, but that’s more complex, and you’d still have to cal the function twice.
  • Because the resulting steps are different, you don’t get caching. Eg, if you run zig build test, and then zig build ci, if the ci re-creates test step by following debug/release matrix, it wouldn’t notice that the test were being run already.
2 Likes
4

With addSystemCommand approach, it is crucial that you also do:

    system_command.stdio = .{ .check = .{} };
    system_command.addCheck(.{ .expect_term = .{ .Exited = 0 } });
    system_command.has_side_effects = true;

Otherwise, they all run sequentially, because

    /// Causes the Run step to be considered to have side-effects, and therefore
    /// always execute when it appears in the build graph.
    /// It also means that this step will obtain a global lock to prevent other
    /// steps from running in the meantime.
    /// The step will fail if the subprocess crashes or returns a non-zero exit code.
    inherit,
2 Likes
5

and, yeah, with that in, addSystemCommand approach is actually not that bad!

2 Likes
6

Would this work?

build.zig:

fn compileExe(b: *Build, optimize: Mode) *CompileStep{
  // This could also be a test step.
  return b.addExecutable(.{
    .target = ...,
    .optimize = optimize,
    // All other stuff
  }); 
}

pub fn build(b: *Build) void{
  var steps: [2]*Step = .{
    &compileExe(b, .Debug).step,
    &compileExe(b, .ReleaseFast).step,
  };

  const step_ci = b.step("ci", "Run full suite of checks");
  for(steps) |step|{
    step_ci.dependOn(step);
  }
}
 
  const optimize = b.getStandardOptimizeOption(.{});
  const install = b.getInstallStep();
  install.dependsOn(switch(optimize){
    .Debug => steps[0],
    else => steps[1],
  });
7

The best way, as tensorusch mentioned, would be to do dependency injection and pass on all configurations you want through some function. If this seems too hard for your case, then you may be interested in this hack where you create a nested ci/build.zig/ci/build.zig.zon:

// ci/build.zig
const std = @import("std");

pub fn build(b: *std.Build) void {
    const install_step = b.getInstallStep();

    const release_modes: []const bool = &.{ false, true };
    for (release_modes) |release_mode| {
        // if you want more targets then you could add another loop

        const something = b.dependency("something", .{
            .release = release_mode,
            .@"example-name" = @as([]const u8, "hello world"),
        });
        install_step.dependOn(&something.builder.top_level_steps.get("test").?.step);
        install_step.dependOn(&something.builder.top_level_steps.get("example").?.step);

        // and if you want to store/save executables somewhere:
        _ = something.artifact("exe");
    }
}


// ci/build.zig.zon
.{
    .name = .ci,
    .version = "0.0.0",
    .fingerprint = 0x3b67f3676f2e7dda, // Changing this has security and trust implications.
    .minimum_zig_version = "0.15.0-dev.151+6e8493daa",
    .dependencies = .{
        .something = .{ .path = "../" },
    },
    .paths = .{ "build.zig", "build.zig.zon" },
}

However, this has quite a few drawbacks because you must do cd ci && zig build. I’m also unsure if the builder.top_level_steps should be used this way from separate builders. Therefore, if possible, I would try to use the original build.zig.

1 Like
8

I need zig build test [-Drelase] working as well.

9

if you abstract the meat of your steps into a function that takes its config as parameters, it should be relatively easy to make a ci step that calls a bunch of configurations

2 Likes
10

That has the problem that there’s no re-use of caches between zig build test and zig build ci.

11

I dont understand how there wouldnt be cache reuse.
can you elaborate?

1 Like
12

Thanks for pushing me on this one! I stand corrected, I had a wrong mental model of how Zig caching works. Basically, for this example:

const std = @import("std");

pub fn build(b: *std.Build) !void {
    const optimize = b.standardOptimizeOption(.{
        .preferred_optimize_mode = .ReleaseSafe,
    });
    const tls_test = b.step("test", "Run tests");
    tls_test.dependOn(test_step(b, optimize));

    const tls_ci = b.step("ci", "Full set of CI checks");
    tls_ci.dependOn(test_step(b, .Debug));
    tls_ci.dependOn(test_step(b, .ReleaseSafe));
}

fn test_step(b: *std.Build, optimize: std.builtin.OptimizeMode) *std.Build.Step {
    const t = b.addTest(.{
        .root_source_file = b.path("test.zig"),
        .optimize = optimize,
    });
    return &b.addRunArtifact(t).step;
}

I was thinking that running zig bulild test and then zig build ci would compile tests in debug mode twice, because there will be separate std.Build.Step.Compile compiles. But it looks (obvious in retrospect) that caching is determined “by value”, not by identity.

That is, the snippet above creates three distinct (in terms of pointer equality) std.Build.Step.Compiles, two of which are equal. So the cacing works as expected:

λ ~/zig-0.14/zig build --summary all test
Build Summary: 3/3 steps succeeded; 1/1 tests passed
test success
└─ run test 1 passed 1ms MaxRSS:1M
   └─ zig test Debug native cached 35ms MaxRSS:36M
matklad@ahab ~/tmp
λ ~/zig-0.14/zig build --summary all ci
Build Summary: 5/5 steps succeeded; 2/2 tests passed
ci success
├─ run test 1 passed 1ms MaxRSS:1M
│  └─ zig test Debug native cached 51ms MaxRSS:36M
└─ run test 1 passed 382ms MaxRSS:1M
   └─ zig test ReleaseSafe native success 4s MaxRSS:288M

Ok, so then my complained is reduced to just “I have to refactor my entire build script”, which is something I can live with, thanks!

5 Likes