How do I access emitted object files in the build system?

I am building a WASM based web app that does text processing. In it, I use Zig to do all my processing and this also involves using the C PCRE2 library for Regex. While I was able to build a command line based PoC for arm64 darwin, I have not had any luck in building my app for WASM using the Zig build system. Since PCRE2 has a build.zig in it, it is fairly trivial to add to my project. Here is a very basic example of my build file:

const std = @import("std");

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});

    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        .name = "pcre2_test",
        .root_source_file = b.path("src/main.zig"),
        .target = target,
        .optimize = optimize,
    });

    const pcre2_dep = b.dependency("pcre2", .{
        .target = target,
        .optimize = optimize,
    });
    exe.linkLibrary(pcre2_dep.artifact("pcre2-8"));

    b.installArtifact(exe);

    const run_cmd = b.addRunArtifact(exe);

    run_cmd.step.dependOn(b.getInstallStep());

    if (b.args) |args| {
        run_cmd.addArgs(args);
    }

    const run_step = b.step("run", "Run the app");
    run_step.dependOn(&run_cmd.step);
}

But when it comes time to build a wasm32-freestanding file, the build system is unable to find some C functions that PCRE2 uses. Functions like malloc, free, and memchr. I have not been able to figure out how to fix this.

So I changed my approach to:

  • Build PCRE2 for WASM using Emcscripten instead of Zig.
  • Use Zig to only build my code to an Object.
  • Use emcc to complete (link?) the last stage.

This ended up working for me and for now I will grudgingly accept that this is my workflow for the time being.

I am now in the process of trying to unify step 2 and 3 in my build.zig and my question is this, if I am building and copying objects to an output directory, how do I reference them in a separate step. Please see my new build.zig file below:

pub fn build(b: *std.Build) void {
    const target = b.resolveTargetQuery(.{
        .cpu_arch = .wasm32,
        .os_tag   = .freestanding,
    });
    const optimize = b.standardOptimizeOption(.{});


    const exe = b.addObject(.{
        .name = "wasm-test",
        .root_source_file = b.path("src/main.zig"),
        .target = target,
        .optimize = optimize,
    });


    exe.addSystemIncludePath(b.path("../emsdk/upstream/emscripten/cache/sysroot/include/"));
    exe.linkSystemLibrary("pcre2-8");
    exe.addIncludePath(b.path("../pcre2/wasm/include"));
    exe.addLibraryPath(b.path("../pcre2/wasm/lib"));

    b.getInstallStep().dependOn(&exe.step);

    const instDir = b.addInstallDirectory(.{
        .source_dir = exe.getEmittedBinDirectory(),
        .install_dir = .prefix,
        .install_subdir = "wasm-obj",
    });

    b.getInstallStep().dependOn(&instDir.step);
}

The instDir step will copy two files wasm-test.o and wasm-test.o.o to my zig-out directory and both these files are needed when I use emcc. I would like to reference them and run the emcc command from within the Zig build process.

So far from what I have tried by using getPath() I see a: getPath() was called on a GeneratedFile that wasn't built yet. error. I am assuming that is because the addObject is just an intermediate step and so nothing is fully finalized yet? Is this even doable or am I missing some steps in between?

Hello @sheran
Welcome to ziggit :slight_smile:

To use the wasm32-freestanding[-musl] C functions link with libc:

    exe.linkLibC();

Thank you @dimdin for the welcome and for your reply!

I am recapping here to see if I follow your advice correctly:

zig version: 0.14.0-dev.1550+4fba7336a

main.zig

const std = @import("std");
const re  = @cImport({
    @cDefine("PCRE2_CODE_UNIT_WIDTH", "8");
    @cInclude("pcre2.h");
});

const PCRE2_ZERO_TERMINATED = ~@as(re.PCRE2_SIZE, 0);

pub export fn trial(pt: [*]u8, ptLen: usize) isize {
    const pattern: re.PCRE2_SPTR8    = "argle";
    var   errornumber: c_int         = undefined;
    var   erroroffset: re.PCRE2_SIZE = undefined;

     const regexp : ?*re.pcre2_code_8 = re.pcre2_compile_8(
        pattern,
        PCRE2_ZERO_TERMINATED,
        0,
        &errornumber,
        &erroroffset,
        null
    );
    defer re.pcre2_code_free_8(regexp);
    if (regexp == null) {
        return 253;
    }

    var haystack : [1024]u8 = undefined;
    std.mem.copyForwards(u8, &haystack, pt[0..ptLen]);
    const subject: re.PCRE2_SPTR8 = &haystack[0];
    const subjLen: re.PCRE2_SIZE  = ptLen;

    const matchData: ?*re.pcre2_match_data_8 = re.pcre2_match_data_create_from_pattern_8(regexp, null);
    defer re.pcre2_match_data_free_8(matchData);
    const rc: c_int = re.pcre2_match_8(
        regexp,
        subject,
        subjLen,
        0,
        0,
        matchData.?,
        null
    );

    if (rc < 0){
        return rc;
    }

    const ovector = re.pcre2_get_ovector_pointer_8(matchData);
    if (rc == 0){
        return 255;
    }

    if(ovector[0] > ovector[1]){
        return 256;
    }
    return @intCast(ovector[0]);
}

build.zig

const std = @import("std");

pub fn build(b: *std.Build) void {
    const target = b.resolveTargetQuery(.{
        .cpu_arch = .wasm32,
        .os_tag   = .freestanding,
    });
    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        .name = "wasm-test",
        .root_source_file = b.path("src/main.zig"),
        .target = target,
        .optimize = optimize,
    });

    const pcre2_dep = b.dependency("pcre2", .{
        .target = target,
        .optimize = optimize,
        .linkage = .static,
    });

    exe.linkLibrary(pcre2_dep.artifact("pcre2-8"));
    exe.linkLibC();
    exe.entry = .disabled;
    exe.rdynamic = true;

    b.installArtifact(exe);

}

build.zig.zon

.{
    .name = "wasm-test",

    .version = "0.0.0",

    .dependencies = .{
        .pcre2 = .{ .url = "https://github.com/PCRE2Project/pcre2/archive/cd4c0e3fc172fd41dd6b74a79dea21fee39cbc43.tar.gz", .hash = "122081c0c1d2bc538034c621b645febbe01c42791d1d2d1ad19dafe785f09b7b818b" },
        .emsdk = .{
            .url = "git+https://github.com/emscripten-core/emsdk#3.1.61",
            .hash = "12200ba39d83227f5de08287b043b011a2eb855cdb077f4b165edce30564ba73400e",
        },
    },

    .paths = .{
        "build.zig",
        "build.zig.zon",
        "src",
    },
}

Then if I do a zig build I’ll see a whole load of error: wasm-ld ... undefined symbol: malloc errors. Here’s a snipped list:

error: wasm-ld: /Users/sheran/Documents/code/wasm-test/.zig-cache/o/5fee642edf2d3dabc21a9fd3b785a4cb/libpcre2-8.a(/Users/sheran/Documents/code/wasm-test/.zig-cache/o/d30151302d95fe74787e6214506218a9/pcre2_context.o): undefined symbol: malloc
error: wasm-ld: /Users/sheran/Documents/code/wasm-test/.zig-cache/o/5fee642edf2d3dabc21a9fd3b785a4cb/libpcre2-8.a(/Users/sheran/Documents/code/wasm-test/.zig-cache/o/d30151302d95fe74787e6214506218a9/pcre2_context.o): undefined symbol: free
error: wasm-ld: /Users/sheran/Documents/code/wasm-test/.zig-cache/o/5fee642edf2d3dabc21a9fd3b785a4cb/libpcre2-8.a(/Users/sheran/Documents/code/wasm-test/.zig-cache/o/e769f3db0275b43dc9b818c383c2920d/pcre2_match.o): undefined symbol: memchr

If I check a file to see if it was built for the correct arch:

file /Users/sheran/Documents/code/wasm-test/.zig-cache/o/e769f3db0275b43dc9b818c383c2920d/pcre2_match.o
/Users/sheran/Documents/code/wasm-test/.zig-cache/o/e769f3db0275b43dc9b818c383c2920d/pcre2_match.o: WebAssembly (wasm) binary module version 0x1 (MVP)

The main.zig code works fine if I do the following:

  1. Build PCRE2 with Emscripten: emconfigure, emmake, emmake install
  2. Build an object using Zig first:
zig build-obj -isystem ../emsdk/upstream/emscripten/cache/sysroot/include/ -I../pcre2/wasm/include -L../pcre2/wasm/lib -lpcre2-8  src/main.zig -target wasm32-freestanding
  1. Use emcc to output the WASM:
emcc main.o main.o.o ../pcre2/wasm/lib/libpcre2-8.a -o final.js --no-entry -sEXPORTED_FUNCTIONS="_trial" -O3 -sSTANDALONE_WASM

Those three steps you listed are what you have to do to target Emscripten, Zig doesn’t come with the Emscripten build tools or the Emscripten libc. You can invoke the emcc binary from your build.zig to automate the build process though.

If you are building for emscripten you should also probably set the operating system as emscripten not freestanding. Then the Zig standard library will actually use the Emscripten libc.

You can probably build the pcre2 library with the Zig C compiler by using the wasm32-emscripten target and adding the include path for the Emscripten libc headers. It doesn’t look like the build.zig in that repo will support that by default though.

You have to use the emcc binary for linking.

Edit: Sorry, I just re-read your initial post and it looks like you were already trying to do what I suggested. I haven’t been using Zig recently, so all of my old Zig Emscripten examples are out of date. Take a look at zig-raylib for one example of using a C library dependency and building for Emscripten.

Thanks, yeah I resigned myself to building with Emscripten because I couldn’t get the build working with plain Zig. I’ll take a look at the raylib mechanism. I quickly glanced at it and it looked like a similar method that @floooh is using in his Pacman game.

I guess that then brings me back to my other question which was:

If b.addObject() emits two object files that I need to then run emcc on, what is the best approach to reference them in the Zig build system as per my build file here:

pub fn build(b: *std.Build) void {
    const target = b.resolveTargetQuery(.{
        .cpu_arch = .wasm32,
        .os_tag   = .freestanding,
    });
    const optimize = b.standardOptimizeOption(.{});


    const exe = b.addObject(.{
        .name = "wasm-test",
        .root_source_file = b.path("src/main.zig"),
        .target = target,
        .optimize = optimize,
    });


    exe.addSystemIncludePath(b.path("../emsdk/upstream/emscripten/cache/sysroot/include/"));
    exe.linkSystemLibrary("pcre2-8");
    exe.addIncludePath(b.path("../pcre2/wasm/include"));
    exe.addLibraryPath(b.path("../pcre2/wasm/lib"));

    b.getInstallStep().dependOn(&exe.step);

    const instDir = b.addInstallDirectory(.{
        .source_dir = exe.getEmittedBinDirectory(),
        .install_dir = .prefix,
        .install_subdir = "wasm-obj",
    });

    b.getInstallStep().dependOn(&instDir.step);
}

Essentially, the two files get put into my zig-out/wasm-obj directory.

I’m not up to date on the Zig build system and there is probably a way to use the object file, but using a static library worked for me in the past (and it seems to still work).

2 Likes

I think you’re looking for:

std.Build.Step.Run.addArtifactArg - Zig Documentation (ziglang.org)

or similar. You can reference generated artifacts for use as arguments to a run step. You’ll create a Run step to run your emcc tool.

Yes all the examples I have seen are structured like this:

  • for non wasm builds create a normal executable and build
  • for wasm builds create a static library instead and then do the link step by invoking emcc (using the wasm32-emscripten target)

I will add my game (that project is on zig 0.13.0) to the list of references (mostly based on @permutationlock’s example zig_emscripten_threads/build.zig at main · permutationlock/zig_emscripten_threads · GitHub).

There is also sokol-zig/build.zig at master · floooh/sokol-zig · GitHub which is interesting because it even downloads emscripten via the build system (and then activates it via emscripten commands), however the way it is done there is a bit hacky (but works), ideally the community would figure out a good way to package emscripten (create a build.zig for it), but it seemed a bit difficult to do that well (at least when you lack practice in creating zig wrappers).

So I opted for the raylib approach of just providing a sysroot parameter for emscripten builds and manually installing emscripten.

2 Likes

Good to get confirmation that the emcc was is the way to go. I will use that approach.

I was actually trying to use the sokol-zig approach! I think I need to go back to the drawing board though, because you and @permutationlock have said static library and I’m using object files. Thanks very much for the additional references!

1 Like