Using stdout generated file as Zig source code in build.zig

I’ve been working on a code generator for Zig to make translated C easier to use, and one of the features I wanted was @import-ing the standard output of a code-generating executable. I’ve looked across the internet for information on how to do this, but it seems like there is no full explanation, yet. This post assumes moderate knowledge of the build system, and is using Zig 0.15.2.

The following (generalized) build script fails:

const std = @import("std");

fn generate_code(b: *std.Build, c: *std.Build.Step.TranslateC) *std.Build.Module {
    const c_module = b.createModule(.{
        .root_source_file = c.getOutput()
    });

    const gen = b.addExecutable(.{
        .name = "generator",
        .root_module = b.createModule(.{
            .target = b.graph.host,
            .optimize = .Debug,
            .root_source_file = b.path("generator.zig"),
        })
    });
    gen.root_module.addImport("C", c_module);
    const run_gen = b.addRunArtifact(gen);
    const source_code = run_gen.captureStdOut();

    const generated_module = b.createModule(.{
        .root_source_file = source_code
    });
    generated_module.addImport("C", c_module);

    return generated_module;
}

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    const translate_c = b.addTranslateC(.{
        .root_source_file = b.path("include/lib_header.h")
    });

    const generated_module = generate_code(b, translate_c);

    const exe = b.addExecutable(.{
        .name = "exe",
        .root_module = b.createModule(.{
            .root_source_file = b.path("src/main.zig"),
            .target = target,
            .optimize = optimize,
        })
    });
    exe.root_module.addImport("lib", generated_module);
    exe.root_module.linkSystemLibrary("lib", .{.needed = true});
    b.installArtifact(exe);
}

The file structure looks like this:

build root
- include/lib_header.h
- src/main.zig
- generator.zig
- build.zig

To sum it up, the steps are:

  1. Translate a C file
  2. Compile and run Generator executable with translated C code
  3. Make module using stdout of Generator as the root_source_file
  4. Create the main executable
  5. Import the generated module into the main executable

It seems like the build would work, but it fails with this nondescript error:

error: no module named 'lib' available within module 'root'

The reason this fails is because capturing the standard output of an executable with .captureStdOut() produces a temporary file named literally “stdout”, which doesn’t end in .zig, so the build system doesn’t recognize it as Zig source code. Creating a module where itsroot_source_file is not Zig code is ok (you can still use @embedFile on it), so the build system doesn’t find anything wrong until the module is @import-ed as Zig code.

To make this work, the generated file needs to be renamed to anything ending in .zig. This is ability is obscured by the build system, probably for good reason, but here is how to do so:

fn generate_code(b: *std.Build, c: *std.Build.Step.TranslateC) *std.Build.Module {
    const c_module = b.createModule(.{
        .root_source_file = c.getOutput()
    });

    const gen = b.addExecutable(.{
        .name = "generator",
        .root_module = b.createModule(.{
            .target = b.graph.host,
            .optimize = .Debug,
            .root_source_file = b.path("generator.zig"),
        })
    });
    gen.root_module.addImport("C", c_module);
    const run_gen = b.addRunArtifact(gen);
    const source_code = run_gen.captureStdOut();
    // NOTE: captureStdOut creates run_gen.captured_stdout, it is null before

    // REQUIRED CHANGE: make the filename end in .zig
    run_gen.captured_stdout.?.basename = "stdout.zig";

    const generated_module = b.createModule(.{
        .root_source_file = source_code
    });
    generated_module.addImport("C", c_module);

    return generated_module;
}

After this, the build works!

Another option would be to use run_gen.addOutputFileArg("out.zig"), which requires the executable to accept an argument for the filepath, and create the file itself. This complicates the implementation slightly, since it needs to interact with I/O to create a new file. Because of the upcoming changes to I/O in Zig 0.16, I wanted to reduce my reliance on the current I/O interface, leading me down this rabbithole.

1 Like

you’re writing to stdout, which needs an Io interface in 0.16 anyway.

1 Like

I’m with @vulpesx on this one. You’re making a file, so just make a file. It’s not like the Unix shell where you’re rewarded for using stdin and stdout: zig build wants you to make a file, which you do anyway by collecting stdout and then renaming it.

The build system is very flexible, so you made it work, but you’ve added an epicycle which you don’t really need, and because you’re coloring outside the lines it’s more likely that refactors to the build system down the road will break what you’re doing.

I suggest just making a file, for the sake of you, later, when you come back to the project and have to figure out what it’s doing.

2 Likes

On master, std.Build.Run.captureStdOut() has been updated to take a second argument that lets you specify the output filename. On 0.15.2, you need to use the captured_stdout.?.basename hack.

If you want to multi-target both 0.15.2 and master, you can use reflection to feature-detect the second argument, like I did here for my OpenGL binding generator.

2 Likes

Well that changes things.

Were it me, I’d make a file, for what may as well be aesthetic reasons. If the task is “write a file”, I would choose “write a file” over “turn stdout into my file”.

But I see no compelling reason not to go this route any longer, in particular:

Is no more likely to apply to this than it is to anything else.

1 Like

Let’s say I have a script that makes a file. How do I place the file in src?

I want to commit the generated file as the generation doesn’t need to happen except once every new moon. How does one do that?

Always look at the build system doc it describes a lot of the more complex usecases quite well:
https://ziglang.org/learn/build-system/#mutating-source

1 Like

this is perfect, thank you. kept browsing the api and didn’t think to look in the article

1 Like