Is there a roadmap / plan for "the right way" to generate files during a build?

Explain
1

Hi all,

I’ve been trying to google and research the best / possible ways of generating code from build.zig which is then used in a later build step. My practical use case is porting a C/C++ project where cmake runs a bash script which generates some files which are then included from one of the .cpp files.

From what I can tell the preferred way to do this in zig is building a small zig program and running that as a build step via something like addRunArtifact() and hooking the inputs and outputs into the build system via addOutputFileArg() / addOutputFileArg. (docs). Doing this seems to require you to pass all input files and and all output files as command line arguments to the executable, which I imagine can become impractical if you’re dealing with a lot of files. Alternatively you need to spawn this executable many times, once per input+output, which also seems not nice.

However, to me it seems a lot less complex and wasteful to simply run a zig function instead of running an executable (potentially lots of times) only to generate a few strings and write a few files. It seems like a bunch of people are creating steps using a custom makeFn function, but if you do that then it seems like it’d be a bunch of work to integrate your custom makeFn into the caching system such that the function is not rerun unecessarily, input files are watched etc. It works, but this kind of custom build step seems to always run, regardless of whether the input files have changed or not. The input files also are not considered by --watch by default.

Various posts like this one from @castholm indicate that the makeFn approach will be made impossible at some point.

There are others like @floooh who have asked for a function-based ways of creating generation steps.

tldr; I guess the thing I’m asking is

  • is it decided that custom makeFn based steps will be deprecated
  • and/or whether there should be a nice API to run a zig function with inputs and outputs instead of spawning executables for file/code generation and a general RFC

Am I overthinking this? Is worrying about executing extra scripts instead of running functions premature optimisation?

2

pointing out, you can pass input and output directories too.

1 Like
3

Yeah, having to write a cmdline tool just for code generation is kind of the worst case and should only be needed when nothing else works.

There’s the boilerplate of cmdline arg parsing (and those args are not typesafe because they need to go through a string conversion), for more complex input args the build.zig may need to write a JSON file which then also needs to be parsed. Other input files may also need to be written, but the content of those files may only be available as the result of other build steps etc etc etc…

IMHO it would be better if the entire build system would be built around the idea of ‘function DAG nodes’, e.g. a node in the build graph is essentially a function pointer, a way to pass input/output args between such nodes, and to connect nodes into a dependency graph, where outputs of depdendency nodes flow into inputs towards the root node.

E.g. each DAG node has:

  • a list of typed inputs
  • a list of typed outputs
  • an (ideally pure and async) function which turns the inputs into outputs
  • a way to connect node outputs to node inputs

Everything else (e.g. the current ‘build steps’) would be built on top of this fundamental ‘function node’, but (and that’s the important part) build.zigs may also ‘derive’ their own node types and hook them into the build system.

The code in those functions must be properly ‘isolated’ so that they can be scheduled in parallel.

Such a system might actually be a nice proof-of-concept of the new IO system :wink:

Bonus points for allowing to inspect the build graph for visualizations.

2 Likes
4

I wonder if it’s possible to use std.zon.Serializer to generate a ZON file at build time, then import that as a pseudo-namespace.
I’ve never tried any kind of build-time codegen, but in my mind if that’s possible it’d be the easiest/nicest way to do it.

2 Likes
5

Outputting zon totally legit and works Great, but does not get around building an exe, or use a WriteFiles step. Reading and Writing files directly from the build runner also breaks the dag system.
You can import the resulting file as a module in a separate .zig file that contains the matching type.

6

As vulpesx mentioned, you can do outputDirectoryArg too. And for the input, there is no inputDirectoryArg yet, but you can just pass in the directory path as a normal arg. The only issue is that the watcher won’t recognize changes for the file inside. But there is a workaround for that too; you just manually iterate over each file and add them as inputFileArg. However, for your actual program, you can ignore those and take the input directory arg (passed as a normal arg).

Here I’ve an example:

https://github.com/nurulhudaapon/zx/blob/main/build.zig#L176-L198

I’m basically parsing .zx (a special extended Zig) on the fly, transpiring them to Zig and then creating executable that uses all that transpired file, all the output files are in the .zig-cache/o directory, because I used addOutputDirectoryArg, and that is good as I don’t want the user of the module to not have to worry about the transpired Zig codes.

7

makeFn will 100% be removed.

The Zig build system build process (and the process for many other make-inspired build systems as well) is divided into two phases:

  • the “configure” phase, which takes names of top-level steps and other options provided by the user as command-line arguments and constructs a graph of the build steps that need to be run in order to produce the desired result without running any steps, and
  • the “make” phase which actually runs all steps determined by the prior phase.

Your build() function corresponds to the configure phase. Currently when you run zig build, Zig compiles and runs an executable from your build.zig that first calls your build() function to construct the graph and then runs all steps in the graph, in the same process.

There are two main problems with this approach. The first and most important problem is that because the build runner executable is just a regular Zig program, it has full access to the host system and can do anything any other arbitrary program can, which obviously has security implications. The second and lesser problem is that needing to re-run the configure phase to construct the build graph every time you run the same zig build command with the same options is a waste of time and resources.

To fix the security issue, the plan is to compile your build() function to WebAssembly and run it in a sandbox without any access to the host system. I believe that the idea is that this will serialize the result of the configure phase to some format (Zon?) which is then read and processed by a single common build runner built into the Zig compiler. This way, no untrusted code that could do naughty things to the host system needs to be executed as part of a simple standard build process.

If you need to generate code as part of your build, you will need to compile an executable. There’s no real way around this, functions can’t be serialized. Presumably the compiler will also provide a way to let users running your build.zig vet all untrusted programs that will need to be executed as part of the build and approve/deny as appropriate.

8 Likes
8

Thanks for the input everyone! I was aware of most of the workaround of ignoring args and adding input directories and while those may often work, I don’t really want to move the logic of what the inputs and outputs are out of build.zig and into the generator-executable, as that would mean (I think) leaking build (graph) logic from build.zig into the generator, which I want to avoid.

I can only echo the boilerplate required for executable-based generators and I hadn’t actually considered type safety for inputs/outputs yet, though I agree with that too. I guess when using generator executables you’d need to serialize the inputs and outputs to .zon/a module to preserve the type information as (I think) was hinted at above. Experimenting with that (input zon/module and output zon/modules) sounds interesting.

makeFn will 100% be removed.

thank you for elaborating @castholm! I’m following the logic around sandboxed configure logic and a common trusted build runner.

But, and I may be missing something, regarding this part

If you need to generate code as part of your build, you will need to compile an executable. There’s no real way around this, functions can’t be serialized.

I’m not sure that follows? If my code/data generator logic (an executable in today’s world) fundamentally takes inputs and generates outputs, why can’t that be made a module / library executed within a sandbox as well? In other words, functions may not be serialized but could be compiled to wasm and hooked into the build graph. I imagine they’d need to be called from the build runner but run in a sandbox, with sandbox boundaries for inputs and outputs. Then we’d have a nice API for typed inputs and outputs while no longer needing to spawn extra executables, hack around with input/output args/boilerplate and input/output serialization etc.

2 Likes
9

I did follow the same process as OP, but I was recently taught an essential upgrade by @MasonRemaley. You can have an executable discover and produce multiple files on its own as long as it also produces a make (?) dependency file. Relevant std.build functions have depFile in the name.

1 Like
10

My comment about vetting untrusted programs is more about things like running shell scripts, or compiling+running programs that need to do advanced tasks. I don’t see why you couldn’t sandbox code generators or other similarly trivial and safe programs.

However, in my view you probably want to run the code generator lazy, not eager, and defer it up until as late as possible in the make phase to avoid doing unnecessary work. You don’t want to run it in the configure phase. In other words, your build.zig produces a graph of steps, one of which is “run my code generator”, and writes that graph to a file. This file is then read by the build runner and only then, if the right conditions are met, runs your generator. The question is then, how does it find your generator?

The build system could probably provide nice APIs for this that handle the boilerplate input/output wrangling for you. E.g. imagine something like this, where you specify the name of a public decl in your build.zig and it will automatically set up everything else (compiling an executable and calling it with the right arguments):

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const file_basename = "file.txt";

    const generated_dir: std.Build.LazyPath =
        b.addFunction("generateCode", .{ target, file_basename });
    const generated_file = generated_dir.path(b, file_basename);
}

pub fn generateCode(
    ctx: std.Build.Function.Context,
    target: std.Target,
    file_basename: []const u8,
) !void {
    const file: std.Io.Writer = try ctx.createFile(file_basename);
    try file.writeAll(...);
}

But I don’t think you can work around using names of public decls reachable from the build.zig. The build system needs to be able to find your function statically, you can’t give it a function pointer determined at runtime (or compile time for that matter) and tell it “compile this”.

2 Likes
11

Quite frankly, there’s currently way more stuff run during configuration phase than it should.

For example downloading dependencies should also only happen during build phase not during configuration phase because downloading them also depends on the target which should be reached not just the configuration, but the target which should be reached is a build phase thing.

2 Likes
12

downloading dependencies happens before the configure phase, it needs to as the root and its dependencies interact with each other during the configure phase.
It is possible to redesign the system so that’s not necessary, but It would be more limiting than what we have now.

This is a usecase for lazy dependencies, they wont be downloaded unless they are referenced in the configure phase, at which point they are downloaded the configure phase is re run for the above reasons.

13

I would say it’s the other way around; the current design for lazy dependencies is deeply flawed and impose a lot of limitations. For example, if a package exports two modules, a and b, and b lazily depends on a different 500 MiB package, there is no clean way for a consumer to only import a without unnecessarily fetching that package (without resorting to awkward workarounds like -Dexport_b=true options).

The only ability we would lose out on with lazy dependencies resolved during the make phase is the ability to @import()/b.lazyImport() them into a build.zig. But I would recommend that large SDK-style projects that are designed to be imported during the configure phase should split their package up into a small eager package that exposes only the build.zig utilities, which lazily depends on the larger brunt of the package. A regular @import() is better developer UX either way; b.lazyImport() is an enormous ugly hack (I can say this confidently because I implemented it).

In an ideal build system all “objects” would work like LazyPath. There should be LazyCompileStep, LazyModule, LazyValue(T) and so on.

2 Likes
14

Yeah, currently because of this I think that using options instead of targets is the better UX, even if it’s incredibly awkward and not at all intuitive since this makes options have a double meaning (defining what should be built and how).