Any example of caching a generated Zig file?

Help
1

I’m currently compiling and running a Zig program to generate a Zig source file and put it in src. Is there a way to take advantage of the build system’s caching to only generate the file if the generating Zig program source changes?

EDIT: I set up a working example project with the solution here

1 Like
2

I think you can do this in the program itself, if you compare the previous hash of the file to the current one. Not sure about using the build system.

3

Yes, that’s precisely how I’m doing it now. I just wanted to see if I could leverage the build system’s caching and not have to do it on my side. If you think about it, it’s pretty much the same situation as compiling a binary. If the source hasn’t changed, you can reuse the cached version.

1 Like
4

I could be wrong, but I think Step.Run.addFileArg (for the input) and Step.Run.addOutputFileArg (for the output) are intended to be usable for this purpose. There’s also addDepFileOutputArg if there are dependencies that are discovered when processing the input.

EDIT: There’s also a extra_file_dependencies field in Step.Run.

1 Like
5

I’m studying the source code now. From what I can tell, this basically means that instead of the generator writing out to a file directly, it should rather write to stdout and then have the build system capture that output?

6

I don’t believe so. I believe the idea would be something like:

var generator_exe = b.addExecutable(...);

// generate step will run <generator> </path/to/input.txt> </path/to/output.zig>
// so generator_exe should use args[1] as the input path
// and args[2] as the output path
var generate_step = b.addRunArtifact(generator_exe);
generate_step.addFileArg(.{ .path = "input.txt" });
const output_file = generate_step.addOutputFileArg("output.zig");

const install_output = b.addInstallFile(output_file, "dest/output.zig");
b.getInstallStep().dependOn(&install_output.step);

EDIT: Here’s where the run step caching for input/ouput file args is handled btw:

2 Likes
7

EDIT: See full working example project here

Ok, thanks to @squeek502 pointing me in the right direction, I think I got it. I created a new project to test this out. I created src/gen.zig that just writes out the typical hello world Zig program to the the path provided via the command line arg passed in by the build system. I then added this to the standard build.zig:

    const gen_exe = b.addExecutable(.{
        .name = "gen_exe",
        .root_source_file = .{ .path = "src/gen.zig" },
        .target = target,
        .optimize = optimize,
    });

    const run_gen_exe = b.addRunArtifact(gen_exe);
    run_gen_exe.step.dependOn(&gen_exe.step);
    const output_file = run_gen_exe.addOutputFileArg("foo.zig");
    if (b.args) |args| run_gen_exe.addArgs(args);

    const gen_write_files = b.addWriteFiles();
    gen_write_files.addCopyFileToSource(output_file, "src/foo.zig");
    b.getInstallStep().dependOn(&gen_write_files.step);

and to the standard exe that comes with the normal build.zig I added the last line shown here:

    const exe = b.addExecutable(.{
        .name = "gen_test",
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = optimize,
    });
    exe.step.dependOn(&run_gen_exe.step); // added this

And with that, when I run zig build run I see src/foo.zig generated. If I run again, the generated file isn’t touched. If I modify src/gen.zig and then run zig build run again, I see src/foo.zig re-generated. Yes! For Great Justice! lol

2 Likes
8

What does this actually do? Does it write out the output (stdout) from the gen step to the given path?

EDIT: Ok, after looking at the source I see that addInstallFile will install the file relative to the install prefix, which if I’m not mistaken by default is zig-out/bin. So I think in my case I wouldn’t need that step. But anyway good to know.

9

@dude_the_builder, what does your gen.zig look like? Does it actually use the output path passed in via the args? My understanding was that addOutputFileArg takes a basename and the actual path ends up being something like zig-cache/tmp/<some hash>/basename, so the install step is necessary to put that file in the correct location. If you’re not using the path from the args in your generator, though, then it’ll be using the wrong file path for determining if there’s a cache hit (it’ll look at the zig-cache version which isn’t actually written to by your generator)

I’m not really sure what the intended way to handle installing into src would be, or if there’s some other intended way to handle this.

1 Like
10

Once again, you’re tottally right. I was writing out the file directly from my gen.zig and not using the command line arg provided by the build system. Now I updated the post above to reflect the working build.zig which now really does use the build system cache for binaries and the generated file. Thanks so much @squeek502 . The build system is incredible!

1 Like
11

New build system guide has an example that avoids the slightly hacky ‘install to ../src/foo.zig’ stuff by using addWriteFiles/addCopyFileToSource instead:

https://ziglang.org/learn/build-system/#mutating-source-files-in-place

4 Likes
12

That’s awesome! Thanks for sharing this. I updated the above example and the example repo.