Running the Zig Fuzzer with C code linked?

Hello everyone.

I’m trying to run the Smith fuzzer on a project, which includes compiled C libraries. But I seem to be stuck. A summary of the journey so far:

I developed the project on 0.16.0. While the --fuzz flag does build, the run immediately dies with a strange crash unrelated to my code. The last time I saw this type of crash, it was an ABI problem. Since fuzzing is a different ABI within Zig, I am guessing this is caused by me opting into LLVM linking instead of the Zig linker.

Since that is just to work around a bug fixed upstream, I made a branch and upgraded the project to current unreleased master (3e15e99e60), and used the Zig linker.

The project builds and passes tests, but when I tried building it with --fuzz, I get a bunch of undefined symbols for the sanitizer library, all of which point back to the C code (very long list of C objects excluded):

error: undefined symbol: __sanitizer_cov_trace_cmp4
error: undefined symbol: __sanitizer_cov_trace_const_cmp2
error: undefined symbol: __sanitizer_cov_trace_switch
...

According to StackOverflow, this is because clang needs a special flag in order to compile and link everything properly. A flag which obviously the Aro compiler was not being passed, and the Zig compiler was generating some other way.

When I went looking for how to pass arbitrary flags to my translate-c build step, I found this comment from Andrew:

It makes sense why this is seen as a missing enhancement, but it is in fact working as designed. The idea is to […] Migrate from this build step to external package dependency, thereby gaining more options. After these migrations, Zig will stop vendoring Aro source code and the build step will be removed.

That sent me looking through the Aro source code. I don’t see anything about sanitizers yet. And this conversion seems intimidating, to say the least.

Hence my post, trying to figure out where do I go from here:

  1. Is there something I can troubleshoot about the SIGABRT crash of the fuzzer on 0.16.0? I’ve always had to blindly guess before.
  2. Is there some well-known build script trick to get linked C code to fuzz, e.g. some linker command that is separate?
  3. Is there anyone who has done the conversion from addTranslateC to the Aro future already who can give me basic guidance?
  4. Is this known not to work and I should just give up?

Any feedback would be helpful.

In 0.16, fuzzing requires:

  1. Release{Safe,Fast,Small} (I have only used ReleaseSafe though)
  2. Compile with llvm (not just linking)
  3. A patch to the standard library’s test runner

For this, you may need to toggle on .sanatize_c = .full on the module with the C library. The library you are linking with may have been compiled with sanatize on, and so when it links those symbols are not provided by the compiler.
https://ziglang.org/documentation/0.16.0/std/#std.zig.SanitizeC
This can be part of addModule options

Interestingly, adding .sanitize_c = .full changes the command line arguments of the Zig’s builder invocation (-fsanitize-c=full now appears), but doesn’t actually change the result. Now I’m wondering if the compiler is doing the right thing and the linker is not somehow…

EDIT: apparently trying to also set .sanitize_c = .off trying to just ignore that code isn’t working either, the symbols are still appearing. So that makes me think I’m not doing this correctly…

This is puzzling me. I don’t know what I’m doing wrong, but .sanitize_c doesn’t seem to do anything?

After running zig init on a new project, all I did was add the amalgamation release of the current stable SQLite, and then add this dependency block to build.zig:

    const sqlite_dep_mod = blk: {
        const tc = b.addTranslateC(.{
            .root_source_file = b.path("sqlite-amalgamation-3530000/sqlite3.h"),
            .target = target,
            .optimize = optimize,
        });
        const sqlite_mod = tc.createModule();
        sqlite_mod.addCSourceFile(.{
            .file = b.path("sqlite-amalgamation-3530000/sqlite3.c"),
            .flags = &.{"-std=c99"},
            .language = .c,
        });
        sqlite_mod.sanitize_c = .off;
        break :blk sqlite_mod;
    };

Then added it as an import to the root module. No code changes at all to main.zig (which helpfully includes a fuzz target already).

And yet, running zig build test --fuzz still gives me linker errors on those symbols, even though I said not to put them in there.

What am I missing?

EDIT: I am apparently missing --release=safe per a previous comment. But unfortunately that doesn’t work for my current project. I’m about to give up on this and try the 0.16.0 method instead…

I have determined that the trouble seems to be caused by zig clang being invoked to compile the C code instead of zig translate-c like I was expecting. Which is why the symbols from libfuzz.a that ships with Zig don’t match.

Unless anyone has any obvious way to force the builder pass to do the other thing, I guess I’ll try going back to the other approach mentioned by @glfmn to get LLVM tooling to do everything.

Will report back how that goes.

I have returned with more info. But unfortunately I think I am confused somewhere, as following all the threads is a bit difficult for me.

First, the simple test project (zig init with very few changes where I did nothing but add the SQLite source code to build.zig) works in ReleaseFast but not in Debug. Since I was suspicious about that, I added a block to the zig boilerplate Smith:

        .add_data => {
            const slice = try list.addManyAsSlice(gpa, smith.value(u4));
            smith.bytes(slice);
            var ptr: ?*sqlite3.struct_sqlite3 = undefined;
            switch (sqlite3.sqlite3_open("/tmp/test.db\x00".ptr, @ptrCast(&ptr))) {
                sqlite3.SQLITE_OK => {
                    _ = sqlite3.sqlite3_close(ptr.?);
                },
                else => {
                    if (ptr) |p| {
                        _ = sqlite3.sqlite3_close(p);
                    }
                    return;
                },
            }
        },

Sure enough, now the linker errors appear in ReleaseFast mode too. I suspected the symbols were just being optimized out before.

Here is my entire build.zig:


const std = @import("std");

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    const sqlite_dep_mod = blk: {
        const tc = b.addTranslateC(.{
            .root_source_file = b.path("sqlite-amalgamation-3530000/sqlite3.h"),
            .target = target,
            .optimize = optimize,
        });
        const sqlite_mod = tc.createModule();
        sqlite_mod.addCSourceFile(.{
            .file = b.path("sqlite-amalgamation-3530000/sqlite3.c"),
            .flags = &.{"-std=c99"},
            .language = .c,
        });
        sqlite_mod.sanitize_c = .full; // I tried off too, no difference
        break :blk sqlite_mod;
    };

    const mod = b.addModule("zig_test", .{
        .root_source_file = b.path("src/root.zig"),
        .target = target,
    });

    const exe = b.addExecutable(.{
        .name = "zig_test",
        .root_module = b.createModule(.{
            .root_source_file = b.path("src/main.zig"),
            .target = target,
            .optimize = optimize,
            .imports = &.{
                .{ .name = "zig_test", .module = mod },
                .{ .name = "sqlite3", .module = sqlite_dep_mod },
            },
        }),
        .use_llvm = true,
    });

    b.installArtifact(exe);

    const run_step = b.step("run", "Run the app");

    const run_cmd = b.addRunArtifact(exe);
    run_step.dependOn(&run_cmd.step);

    run_cmd.step.dependOn(b.getInstallStep());

    const mod_tests = b.addTest(.{
        .root_module = mod,
        .use_llvm = true,
    });

    const run_mod_tests = b.addRunArtifact(mod_tests);

    const exe_tests = b.addTest(.{
        .root_module = exe.root_module,
        .use_llvm = true,
    });

    const run_exe_tests = b.addRunArtifact(exe_tests);

    const test_step = b.step("test", "Run tests");
    test_step.dependOn(&run_mod_tests.step);
    test_step.dependOn(&run_exe_tests.step);
}

I tried to follow @Southporter 's guidance above, but it seems to have made no difference. I did patch the stdlib even though that’s not shown.

I also tried ReleaseSafe for no difference.

I am not sure if I am “Compiling with llvm (not just linking)” correctly, but I do notice that the libfuzz.a that comes with zig does contain a number of __trace functions, but objdump on it shows it does not contain the missing ones from error messages. I suspect that means I am not doing it right.

Any guidance on what I’m missing?

Would you mind sharing your fuzz test so I can mess with it locally?

This might be an interesting thread to follow. You can run zig build --verbose test and it will output all the commands it’s running. It may be linking a different libfuzz.a which you can check with the output.

It does look like you are setting the sanatize_c correctly and using llvm.

To add a bit of clarity, since I think I might be zooming in on the problem…

While --verbose does not show anything that sticks out, --verbose-link show what I am wondering if it is the mismatch. The final link of the fuzzed binary adds in: /home/censored/.cache/zig/o/7c12a15d52aa063ca7b3d2ac87155f55/libfuzzer.a – which I’ll get back to in a moment.

Theorizing that the Zig toolchain was just missing something, I manually added libclang_rt.fuzzer_no_main-x86_64.a from my system path to the end of the command.

The result: the symbols I’m looking for were in there, so the undefined symbol errors are gone. Aha!

Instead I now have duplicate symbols for e.g. __sanitizer_cov_pcs_init – which is not surprising I guess, since I am effectively linking libfuzz twice.

So I then tried just the Zig library out of the command line. That gives me undefined symbols like fuzzer_start_test – which I recognize from Zig’s own stdlib.

So what seems to be happening is, that cached libfuzzer.a artifact is being built incorrectly. But I am puzzled over where it is coming from.

When I clear out all the caches and add --verbose-cc as well as --verbose I don’t see it being produced. I see the SQLite library being run through zig clang so that seems correct… but I don’t see where libcompiler_rt or libfuzzer artifacts are actually compiled. They just come from libfuzzer_zcu.o and libcompilert_rt_zcu.o which are already in cache directories somehow.

I feel like I’m close, but I don’t see any --verbose option that could catch them. Any suggestions?

I did a little digging in the zig repo, and found this issue: https://codeberg.org/ziglang/zig/issues/31412

Seems to be related to what you are seeing. There is a diff on that issue for patching lib/fuzzer.zig that might help.

It took me longer than it should have to get 0.16.0 to build from source given linker bugs on my libc (Docker to the rescue), and then took my best guess at applying the patch to 0.16 (it did not apply cleanly).

It seems to compile now, so I’m pretty sure that was it! But now crashes inside the fuzzer code I patched immediately, which makes me suspect I did the patch wrong…

thread 841 panic: pc counters length and pcs length do not match (169827 != 165308)
/usr/zig/lib/zig/fuzzer.zig:226:56: 0x1a2301c in init (fuzzer)
        if (self.pc_counters.len != pc_table.len) panic(
                                                       ^
/usr/zig/lib/zig/compiler/test_runner.zig:67:29: 0x195a6a7 in main (test)
        fuzz_abi.fuzzer_init(.fromSlice(cache_dir));
                            ^
/usr/zig/lib/zig/std/start.zig:699:88: 0x19564f4 in callMain (test)
    if (fn_info.params[0].type.? == std.process.Init.Minimal) return wrapMain(root.main(.{

The patch I did:

diff --git a/lib/fuzzer.zig b/lib/fuzzer.zig
index c50ce759ba..b6a954a059 100644
--- a/lib/fuzzer.zig
+++ b/lib/fuzzer.zig
@@ -55,6 +55,12 @@ fn bitsetUsizes(elems: usize) usize {
     return math.divCeil(usize, elems, @bitSizeOf(usize)) catch unreachable;
 }
 
+/// https://github.com/llvm/llvm-project/blob/6121df77a781d2e6f9a8e569aa45ccfffd6c7e0e/compiler-rt/lib/fuzzer/FuzzerTracePC.h#L122
+const PCTableEntry = extern struct {
+    PC: usize,
+    PCFlags: usize,
+};
+
 const Executable = struct {
     /// Tracks the hit count for each pc as updated by the test's instrumentation.
     pc_counters: []u8,
@@ -68,7 +74,7 @@ const Executable = struct {
 
     fn getCoverageMap(
         cache_dir: Io.Dir,
-        pcs: []const usize,
+        pc_table: []const PCTableEntry,
         pc_digest: u64,
     ) []align(std.heap.page_size_min) volatile u8 {
         const file_name = std.fmt.hex(pc_digest);
@@ -101,10 +107,10 @@ const Executable = struct {
 
         comptime assert(abi.SeenPcsHeader.trailing[0] == .pc_bits_usize);
         comptime assert(abi.SeenPcsHeader.trailing[1] == .pc_addr);
-        const pc_bitset_usizes = bitsetUsizes(pcs.len);
+        const pc_bitset_usizes = bitsetUsizes(pc_table.len);
         const coverage_file_len = @sizeOf(abi.SeenPcsHeader) +
             pc_bitset_usizes * @sizeOf(usize) +
-            pcs.len * @sizeOf(usize);
+            pc_table.len * @sizeOf(usize);
 
         var populate: bool = false;
         const size = coverage_file.length(io) catch |e|
@@ -137,11 +143,11 @@ const Executable = struct {
             header.* = .{
                 .n_runs = 0,
                 .unique_runs = 0,
-                .pcs_len = pcs.len,
+                .pcs_len = pc_table.len,
             };
             @memset(trailing_bitset, 0);
-            for (trailing_addresses, pcs) |*cov_pc, slided_pc| {
-                cov_pc.* = fuzzer_unslide_address(slided_pc);
+            for (trailing_addresses, pc_table) |*cov_pc, entry| {
+                cov_pc.* = fuzzer_unslide_address(entry.PC);
             }
             io_map.write(io) catch |e|
                 panic("failed to write memory map of '{s}': {t}", .{ &file_name, e });
@@ -151,12 +157,12 @@ const Executable = struct {
                 .{ &file_name, e },
             );
         } else { // Check expected contents
-            if (header.pcs_len != pcs.len) panic(
+            if (header.pcs_len != pc_table.len) panic(
                 "incompatible existing coverage file '{s}' (differing pcs length: {} != {})",
-                .{ &file_name, header.pcs_len, pcs.len },
+                .{ &file_name, header.pcs_len, pc_table.len },
             );
-            for (0.., header.pcAddrs(), pcs) |i, cov_pc, slided_pc| {
-                const pc = fuzzer_unslide_address(slided_pc);
+            for (0.., header.pcAddrs(), pc_table) |i, cov_pc, entry| {
+                const pc = fuzzer_unslide_address(entry.PC);
                 if (cov_pc != pc) panic(
                     "incompatible existing coverage file '{s}' (differing pc at index {d}: {x} != {x})",
                     .{ &file_name, i, cov_pc, pc },
@@ -201,15 +207,15 @@ const Executable = struct {
             break :blk pc_counters_start[0 .. pc_counters_end - pc_counters_start];
         };
 
-        const pcs = blk: {
-            const pcs_start_name = section_start_prefix ++ "__sancov_pcs1";
-            const pcs_start = @extern([*]usize, .{
+        const pc_table = blk: {
+            const pcs_start_name = section_start_prefix ++ "__sancov_pcs";
+            const pcs_start = @extern([*]PCTableEntry, .{
                 .name = pcs_start_name,
                 .linkage = .weak,
             }) orelse panic("missing {s} symbol", .{pcs_start_name});
 
-            const pcs_end_name = section_end_prefix ++ "__sancov_pcs1";
-            const pcs_end = @extern([*]usize, .{
+            const pcs_end_name = section_end_prefix ++ "__sancov_pcs";
+            const pcs_end = @extern([*]PCTableEntry, .{
                 .name = pcs_end_name,
                 .linkage = .weak,
             }) orelse panic("missing {s} symbol", .{pcs_end_name});
@@ -217,22 +223,22 @@ const Executable = struct {
             break :blk pcs_start[0 .. pcs_end - pcs_start];
         };
 
-        if (self.pc_counters.len != pcs.len) panic(
+        if (self.pc_counters.len != pc_table.len) panic(
             "pc counters length and pcs length do not match ({} != {})",
-            .{ self.pc_counters.len, pcs.len },
+            .{ self.pc_counters.len, pc_table.len },
         );
 
         self.pc_digest = digest: {
             // Relocations have been applied to `pcs` so it contains runtime addresses (with slide
             // applied). We need to translate these to the virtual addresses as on disk.
             var h: std.hash.Wyhash = .init(0);
-            for (pcs) |pc| {
-                const pc_vaddr = fuzzer_unslide_address(pc);
+            for (pc_table) |entry| {
+                const pc_vaddr = fuzzer_unslide_address(entry.PC);
                 h.update(@ptrCast(&pc_vaddr));
             }
             break :digest h.final();
         };
-        self.shared_seen_pcs = getCoverageMap(cache_dir, pcs, self.pc_digest);
+        self.shared_seen_pcs = getCoverageMap(cache_dir, pc_table, self.pc_digest);
 
         return self;
     }
diff --git a/src/Compilation.zig b/src/Compilation.zig
index 695dd8918f..00c528f09c 100644
--- a/src/Compilation.zig
+++ b/src/Compilation.zig
@@ -6600,10 +6600,6 @@ fn addCommonCCArgs(
                     if (san_arg.items.len == 0) try san_arg.appendSlice(arena, prefix);
                     try san_arg.appendSlice(arena, "thread,");
                 }
-                if (mod.fuzz) {
-                    if (san_arg.items.len == 0) try san_arg.appendSlice(arena, prefix);
-                    try san_arg.appendSlice(arena, "fuzzer-no-link,");
-                }
                 // Chop off the trailing comma and append to argv.
                 if (san_arg.pop()) |_| {
                     try argv.append(san_arg.items);
@@ -6638,8 +6634,12 @@ fn addCommonCCArgs(
                     }
                 }
 
+                // This logic must be kept in sync with the coverage emitted by the LLVM backend.
+
                 if (comp.config.san_cov_trace_pc_guard) {
                     try argv.append("-fsanitize-coverage=trace-pc-guard");
+                } else if (mod.fuzz) {
+                    try argv.append("-fsanitize-coverage=edge,inline-8bit-counters,pc-table");
                 }
             }
 
diff --git a/src/codegen/llvm.zig b/src/codegen/llvm.zig
index 1ba3b272da..5dbe565377 100644
--- a/src/codegen/llvm.zig
+++ b/src/codegen/llvm.zig
@@ -1069,7 +1069,10 @@ pub const Object = struct {
             .bitcode_filename = null,
 
             // `.coverage` value is only used when `.sancov` is enabled.
-            .sancov = options.fuzz or comp.config.san_cov_trace_pc_guard,
+            .sancov = comp.config.san_cov_trace_pc_guard,
+            // At least one of these values has to be non-default if `sancov` is
+            // set, otherwise LLVM will implicitly set `TracePCGuard`!
+            // https://github.com/llvm/llvm-project/pull/106464
             .coverage = .{
                 .CoverageType = .Edge,
                 // Works in tandem with Inline8bitCounters or InlineBoolFlag.
@@ -1089,8 +1092,7 @@ pub const Object = struct {
                 // Zig emits its own PC table instrumentation.
                 .PCTable = false,
                 .NoPrune = false,
-                // Workaround for https://github.com/llvm/llvm-project/pull/106464
-                .StackDepth = true,
+                .StackDepth = false,
                 .TraceLoads = false,
                 .TraceStores = false,
                 .CollectControlFlow = false,
@@ -1466,7 +1468,7 @@ pub const Object = struct {
 
             break :f .{
                 .counters_variable = counters_variable,
-                .pcs = .empty,
+                .pc_table = .empty,
             };
         };
 
@@ -1516,18 +1518,14 @@ pub const Object = struct {
         llvm_function.setAttributes(try attributes.finish(&o.builder), &o.builder);
 
         if (fg.fuzz) |*f| {
-            {
-                const array_llvm_ty = try o.builder.arrayType(f.pcs.items.len, .i8);
-                f.counters_variable.ptrConst(&o.builder).global.ptr(&o.builder).type = array_llvm_ty;
-                const zero_init = try o.builder.zeroInitConst(array_llvm_ty);
-                try f.counters_variable.setInitializer(zero_init, &o.builder);
-            }
-
-            const array_llvm_ty = try o.builder.arrayType(f.pcs.items.len, .ptr);
-            const init_val = try o.builder.arrayConst(array_llvm_ty, f.pcs.items);
-            // Due to error "members of llvm.compiler.used must be named", this global needs a name.
+            const cntrs_array_llvm_ty = try o.builder.arrayType(f.pc_table.items.len / 2, .i8);
+            f.counters_variable.ptrConst(&o.builder).global.ptr(&o.builder).type = cntrs_array_llvm_ty;
+            const zero_init = try o.builder.zeroInitConst(cntrs_array_llvm_ty);
+            try f.counters_variable.setInitializer(zero_init, &o.builder);
+            const pc_table_llvm_ty = try o.builder.arrayType(f.pc_table.items.len, .ptr);
+            const init_val = try o.builder.arrayConst(pc_table_llvm_ty, f.pc_table.items);
             const anon_name = try o.builder.strtabStringFmt("__sancov_gen_.{d}", .{o.used.items.len});
-            const pcs_variable = try o.builder.addVariable(anon_name, array_llvm_ty, .default);
+            const pcs_variable = try o.builder.addVariable(anon_name, pc_table_llvm_ty, .default);
             try pcs_variable.setInitializer(init_val, &o.builder);
             pcs_variable.setMutability(.constant, &o.builder);
             pcs_variable.setSection(switch (target.ofmt) {
diff --git a/src/codegen/llvm/FuncGen.zig b/src/codegen/llvm/FuncGen.zig
index f1ebab4443..44568e7834 100644
--- a/src/codegen/llvm/FuncGen.zig
+++ b/src/codegen/llvm/FuncGen.zig
@@ -92,10 +92,10 @@ fn maybeMarkAllowZeroAccess(self: *FuncGen, info: InternPool.Key.PtrType) void {
 
 pub const Fuzz = struct {
     counters_variable: Builder.Variable.Index,
-    pcs: std.ArrayList(Builder.Constant),
+    pc_table: std.ArrayList(Builder.Constant),
 
     fn deinit(f: *Fuzz, gpa: Allocator) void {
-        f.pcs.deinit(gpa);
+        f.pc_table.deinit(gpa);
         f.* = undefined;
     }
 };
@@ -183,7 +183,7 @@ pub fn genBody(self: *FuncGen, body: []const Air.Inst.Index, coverage_point: Air
     switch (coverage_point) {
         .none => {},
         .poi => if (self.fuzz) |*fuzz| {
-            const poi_index = fuzz.pcs.items.len;
+            const poi_index = fuzz.pc_table.items.len;
             const base_ptr = fuzz.counters_variable.toValue(&o.builder);
             const ptr = try self.ptraddConst(base_ptr, poi_index);
             const one = try o.builder.intValue(.i8, 1);
@@ -195,7 +195,7 @@ pub fn genBody(self: *FuncGen, body: []const Air.Inst.Index, coverage_point: Air
             else
                 try o.builder.blockAddrConst(self.wip.function, self.wip.cursor.block);
             const gpa = self.gpa;
-            try fuzz.pcs.append(gpa, pc);
+            try fuzz.pc_table.append(gpa, pc);
         },
     }
     for (body, 0..) |inst, i| {

Any final ideas? I’ll probably just consider this solved as close enough if not.