Writers Block Adding New Build System Feature

Help
1

A couple months back I decided to take a crack at a implementing JSON compilation database support into Zig’s build system (compile_commands.json) since it seemed useful and there was a dead PR from years ago that got part of the way there. See my WIP PR for some context/history.

See this for some info on the artifact I’m trying to generate, but the general idea is as follows:

  • Compilation calls to clang with the flag -MJ[path/to/artifact.json] for a given C/C++ file spit out a JSON fragment that looks something like:
{
 "directory": "/home/user/llvm/build",
 "arguments": ["/usr/bin/clang++", "-Irelative", "-DSOMEDEF=With spaces, quotes and \\-es.", "-c", "-o", "file.o", "file.cc"],
 "file": "file.cc"
},

For better or worse, lots of tooling has been built around these files to help IDEs understand how a file was compiled so it can intelligently do syntax highlighting that’s dependent on #defines, etc.

I’ve made good progress generating these files for individual Step.Compile steps. For instance, build.zig snippet for an executable with some C files:

const exe = b.addExecutable(.{
    .name = "cdb_exe",
    .root_module = b.createModule(
        .{
            .root_source_file = b.path("main.zig"),
            .target = b.graph.host,
            .optimize = optimize,
        },
    ),
});
exe.root_module.addCSourceFile(.{
    .file = b.path("foo.c"),
    .flags = &.{"-Wall"},
});
exe.root_module.addCSourceFiles(.{
    .files = &.{ "bar.c", "baz.c" },
    .flags = &.{ "-Wall", "-Werror" },
});
const cc_exe = exe.getCompileCommandsJson();

Where cc_exe is the LazyPath of the generated compilation db the user can do with what they please (add to install artifacts, perform further processing, etc.). In my branch on that PR, see test/standalone/compdb_single for a working example of how to use this new feature.

However, generally when people want to generate a compile_commands.json, it’s for IDE tooling. So what they really want is a single file that encompasses every compiled C/C++ source file across every invocation of the compiler when you build your project. For instance, if you’re compiling one source file, but into two different artifacts (exe, lib, etc.) with different compiler args, you would expect two entries in your database:

// You compile it with some_arg1 for exe1
{
 "directory": "/home/user/llvm/build",
 "arguments": ["/usr/bin/clang++", "some_arg1", "-c", "-o", "file.o", "file.cc"],
 "file": "file.cc"
},
// You compile it with some_arg2 for exe2
{
 "directory": "/home/user/llvm/build",
 "arguments": ["/usr/bin/clang++", "some_arg2", "-c", "-o", "file.o", "file.cc"],
 "file": "file.cc"
}

Although it’s a little odd to have every compilation command for every artifact in one “file”, this matches how current build systems (CMake) do it.

This is the part I’m stuck on. Mechanically what needs to happen is:

  • The user can get access to a top level merged compilation database LazyPath via const merged_db = b.getCompileCommandsJson();
  • This creates some sort of top level maker step let’s call MergeAllCompdbFiles
  • Every time the user adds a Step.Compile to their build graph (either directly or through including a dependency that has Compile steps), the some_compile_step.getCompileCommandsJson() LazyPath is added as a dependency to MergeAllCompdbFiles
  • After all Maker.Step.Compile complete, MergeAllCompdbFiles Maker step runs generating the overall merged compilation db

Anybody have a good starting point or suggestions for where this quasi-global behavior can get added? I actually had this behavior implemented pre build system rework where I kept track of this top level step (and its dependencies) in the std.Build struct. Now that Configure and Maker phases are separated this approach no longer works, although arguably it was a bit hacky to begin with.

1 Like
2

is it fine to have each step just append to the file? (or perhaps better, to a string that is eventually flushed to the file?)

1 Like
3

i snooped around, is there a reason not to add this global state to the maker struct? where after executing the graph it generates the complete compile commands (if requested).

4

Adding -MJ support to each Step.Compile is already valuable enough IMO. I don’t mind implementing the merging step in my build script.

(In fact, if Step.Compile.getCompileCommandsJson() returns the concatenated fragments without the [...] delimiters, I think it would be easier to stitch multiple compile commands JSON files together and just add those delimiters once, not having to strip before merging)

5

I nerd snipped myself into trying a poc (I didn’t get very far), but I better understand the issue now:

Depending implicitly on all compile steps is troublesome.

It wasn’t clear if you intend to merge the compile db for just the package (and maybe its dependencies too) that ore quests it, or for the entire build graph.
Both have the aforementioned issue, but the latter has the additional issue of dependencies referring to a generated file from the root.

There needs to be some special casing somewhere. The best solution I came up with is similar to what you tried before (which I think could still work), the difference I suggest is to only merge a caller given set of artifacts compile db’s instead of all of them.

This neatly uses the existing systems to solve the issues, dependencies are explicit and properly defined, and it allows the developer to have better control. And yeah, it is ultimately a convenience feature that they could implement themselves.

Edit: I have a working implementation I’ll send a PR for your PR if you want it, but I do need to improve the test (I just copied and modified your existing one) as it can certainly be fooled