Invalidate cache if a file is missing

Help
1

My build script generates some files that get loaded into my application at build/compile time with @embedFile. I need to keep the files accessible to some other system tools as well, so they get generated in the source tree instead of directly in the cache.

To ensure caching works, the generating Zig exe works like this:

  1. Check if the file exists
  2. If it does not, generate it
  3. Copy the file to a addOutputFileArg-added cache location

The problem: If I manually delete the file, I still have it stored in Zigā€™s cache, because the build system does not know about the ā€˜realā€™ location of the generated files.

Is there a way for me to either

  1. Tell the build system that this file is an output of the step, without relying on a generated path like addOutputFileArg

or

  1. Invalidate the cache for the generating step manually by checking if the file is missing

ā€¦ or some other option I havenā€™t thought of. :slight_smile:

2

Unless Iā€™m misunderstanding, Mutating source files in place from the Zig build system guide should work.

3 Likes
3

Could you maybe generate the files in the Cache and add an installFile[WithDir] step for the relevant files to copy them to some other directory for your other tools? Or would this only be able but stuff in the actual zig-build directory?

4

That does seem to be the exact thing I need.

Thank you for the link.

Will mark solved when I get it working. :slight_smile:

1 Like
5

After a little digging around the standard library to see what the functionality is called in the current build API, hereā€™s the solution I ended up with:

// Create build step and file path
const update_generated = b.step("update-generated", "Generate or update required build files");
const generated_file_path_raw = "generated/filename.bin"; // relative to build root
const generated_file_path = b.path(generated_file_path_raw);

// Compile and run the file generator
const generator_mod = b.createModule(.{
    .root_source_file = b.path("generator/main.zig"),
    .target = b.graph.host,
    .optimize = optimize,
});

const generator_exe = b.addExecutable(.{
    .name = "generator",
    .root_module = generator_mod,
});

const generator = b.addRunArtifact(generator_exe);
const cached_file = generator.addOutputFileArg("generated");

// Copy the file from cache to source tree
const usf = b.addUpdateSourceFiles();
usf.addCopyFileToSource(cached_file, generated_file_path_raw);

// Update build graph
update_generated.dependOn(&usf.step);

// < --------------- snip ------------------------ >

// make filename.bin available for @embedFile usage
some_module.addAnonymousImport("filename", .{
    .root_source_file = generated_file_path,
});

Running zig build building some_module now fails if the file generated/filename.bin doesnā€™t exist, until you run zig build update-generated, which creates the file.

The documentation @squeek502 linked mentions you shouldnā€™t do this as part of the normal build process, since it can mess up the cache, so I havenā€™t - this is acceptable behaviour for my use case.

Hope the solution is useful to someone else. :slight_smile:

Edit: I was a tad quick declaring this a solution.

The code does what it says on the tin, so Iā€™m leaving the example as is. But this only fetches the files required by the rest of the build system from cache if theyā€™re missing, it doesnā€™t actually rerun the step, which happens to be important in this case. So itā€™s only half the solution to my problem.

If the generated files exist in the source tree, the cached version is fine. If it is missing from the source tree, the step needs to be rerun, regardless of cache state.

How can I tell Zig this?

6

Just to ensure this is not an XY problem: are these tools called from some step in build.zig? Can these tools accept a file path as an argument? If so,

const tool = b.addSystemCommand(...);

creates a Run step, and

tool.addFileArg(caches_file)

adds the generated file path in cache. In this case, copy to the source tree is not required.

7

They are not called from a build step, no.

The step I want to cache is generating some mock data for testing an embedded Linux application, which gets loaded directly into the application with @embedFile.

A copy of the data needs to be accessible on the host machine to verify the test results, and the tool in question expects the data available as files. The test harness itself is not built or run with Zig, and rewriting the test tool in Zig is a bit out of scope.

Well, at the moment. :wink:

FWIW, itā€™s not blocking for the project. Itā€™s just a minor annoyance Iā€™m wondering if I can massage the build system into fixing, or this is something thatā€™ll be fixed when we (eventually) start porting our test suite to Zig as well.

8

You could just start with running the existing test suite as a system command from your build.zig, if you already run tests I imagine there should be some way to instead run zig build test-suite command which then calls the previous command and passes it some of the paths. Seems like the easiest option for bridging the gap.

Another option would be to actually install the files into the zig-out folder and then point your test-suite at that folder.

It is unclear to me how you generate your file, you only describe the outputs, but what is the input for generating the file? If the file is generated based on some other file, you should declare that other file as an input file to the generation step, that way you would have automatic dependency tracking, where the file gets re-generated when the input file changes.

Without something like that the build system canā€™t know when it should re-generate your file, because it is unclear how it is generated and when it needs to be re-generated. For example if the generator hides its inputs by hardcoding them into the executable or by doing its own requests or side-effects.

Can you describe a bit more how your generator works?

1 Like
9

Another option would be to actually install the files into the zig-out folder and then point your test-suite at that folder.

I ended up going with this as the solution. A step that generates the files installed to one prefix with zig build generate_files --prefix /path/to/files, the rest of the build installed to the default zig-out folder.

Can you describe a bit more how your generator works?

Took it up with my boss and they said no, so unfortunately not.
Such is life in proprietary software, I suppose.

1 Like
10

Well in abstract non-specific terms would have been fine, but if you have solved it, then it doesnā€™t matter anymore. What I meant was describing how it generates, for example reading some input file and then using that to generate something, not what it actually does.

Good that using a different prefix path works for you.

11

Hereā€™s an example straight from zg, which will automatically run the generating code when needed.

build.zig:

    const gencat_gen_exe = b.addExecutable(.{
        .name = "gencat",
        .root_source_file = b.path("codegen/gencat.zig"),
        .target = b.graph.host,
        .optimize = .Debug,
    });
    const run_gencat_gen_exe = b.addRunArtifact(gencat_gen_exe);
    const gencat_gen_out = run_gencat_gen_exe.addOutputFileArg("gencat.bin.z");

   // General Category
    const gencat = b.addModule("GeneralCategories", .{
        .root_source_file = b.path("src/GeneralCategories.zig"),
        .target = target,
        .optimize = optimize,
    });
    gencat.addAnonymousImport("gencat", .{ .root_source_file = gencat_gen_out });

Hereā€™s a snippet for how you use the output file arg ā€œgencat.bin.zā€, within (in this case) codegen/gencat.zig:

    var args_iter = try std.process.argsWithAllocator(allocator);
    defer args_iter.deinit();
    _ = args_iter.skip();
    const output_path = args_iter.next() orelse @panic("No output file arg!");

    const compressor = std.compress.flate.deflate.compressor;
    var out_file = try std.fs.cwd().createFile(output_path, .{});
    defer out_file.close();

You may not need a compressor here, but thatā€™s the general technique.

Shoutout to @dude_the_builder who wrote all this stuff, in addition to founding Ziggit.

1 Like