Errors when trying to run std.testing.fuzz on 0.16

Help
1

I wrote a very simple test, which I wanted to have to verify the serialize → deserialize roundtrip and it seems like the perfect use-case for fuzzing. However, something is off with running the fuzzer as I can’t get it to work.

Test code:

    try std.testing.fuzz({}, struct {
        fn fuzzRoundtrip(_: void, smith: *std.testing.Smith) anyerror!void {
            const base: Header = .{
                .field = smith.value(u16),
            };
            var buf: [2]u8 = undefined;
            var reader: std.Io.Reader = .fixed(&buf);
            var writer: std.Io.Writer = .fixed(&buf);

            try base.write(&writer);
            const clone: Header = try .read(&reader);

            try std.testing.expectEqual(base, clone);
        }
    }.fuzzRoundtrip, .{});

(don’t mind much the header, it’s been redacted for the post);

If I try to run zig test -ffuzz ./path/to/test.zig

/usr/lib/zig/compiler/test_runner.zig:566:55: error: expected type '*const debug.StackTrace', found '*builtin.StackTrace'
                            std.debug.writeStackTrace(trace, stderr) catch break :p;
                                                      ^~~~~
/usr/lib/zig/compiler/test_runner.zig:566:55: note: pointer type child 'builtin.StackTrace' cannot cast into pointer type child 'debug.StackTrace'
/usr/lib/zig/std/builtin.zig:11:24: note: struct declared here
pub const StackTrace = struct {
                       ^~~~~~
/usr/lib/zig/std/debug.zig:604:24: note: struct declared here
pub const StackTrace = struct {
                       ^~~~~~
/usr/lib/zig/std/debug.zig:824:28: note: parameter type declared here
pub fn writeStackTrace(st: *const StackTrace, t: Io.Terminal) Writer.Error!void {
                           ^~~~~~~~~~~~~~~~~
referenced by:
    fuzz__anon_44003: /usr/lib/zig/compiler/test_runner.zig:589:41
    fuzz [inlined]: /usr/lib/zig/std/testing.zig:1232:32
    test.roundtrip: lib/run/Format.zig:45:25

So I assume there’s something between builtin and debug versions of stactrace, so I can force a non-debug build.

If I try to run zig test -OReleaseSafe -ffuzz ./path/to/test.zig

thread 194287 panic: missing --cache-dir=[path] argument
/usr/lib/zig/compiler/test_runner.zig:66:48: 0x10c8fc2 in main (test)
        const cache_dir = opt_cache_dir orelse @panic("missing --cache-dir=[path] argument");
                                               ^
/usr/lib/zig/std/start.zig:699:88: 0x10c784a in callMain (test)
    if (fn_info.params[0].type.? == std.process.Init.Minimal) return wrapMain(root.main(.{
                                                                                       ^
/usr/lib/zig/std/start.zig:190:5: 0x10c731d in _start (test)
    asm volatile (switch (native_arch) {
    ^
error: the following test command terminated with signal ABRT:
.zig-cache/o/ed0c89cb3518ddec370412a2d82854d5/test --seed=0xbbdcead1

Dead end, seems like I’ll have to use zig build test for that.

If I try to run zig build test --release=safe

Just for good measure, I added this on build.zig:

    const lib = b.addModule("mylib", .{
        .root_source_file = b.path("lib/root.zig"),
        .target = target,
        .optimize = optimize,
        .fuzz = optimize != .Debug,
    });

Then the build outcome is this:

error: 'run.Format.test.roundtrip' terminated with signal ABRT with stderr:
       Segmentation fault at address 0xb4
       /usr/lib/zig/fuzzer.zig:904:46: 0x1141488 in ensureCorpusLoaded (fuzzer)
               if (t.start_mut_corpus != math.maxInt(u32)) return;
                                                    ^
       /usr/lib/zig/compiler/test_runner.zig:592:35: 0x10d39ed in fuzz__anon_34598 (test)
               fuzz_abi.fuzzer_start_test();
                                         ^
       /usr/lib/zig/compiler/test_runner.zig:140:68: 0x10c55c6 in mainServer (test)
                       const status: TestResults.Status = if (test_fn.func()) |v| s: {
                                                                          ^
       /usr/lib/zig/compiler/test_runner.zig:71:26: 0x10c5ee3 in main (test)
               return mainServer(init) catch |err| panic("internal test runner failure: {t}", .{err});
                                ^
       /usr/lib/zig/std/start.zig:699:88: 0x105689a in callMain (test)
           if (fn_info.params[0].type.? == std.process.Init.Minimal) return wrapMain(root.main(.{
                                                                                              ^
       /usr/lib/zig/std/start.zig:190:5: 0x105636d in _start (test)
           asm volatile (switch (native_arch) {
           ^
failed command: ./.zig-cache/o/ea2e33e542d291e2773870d3bd43d89d/test --cache-dir=./.zig-cache --seed=0xe42bf7da --listen=-

Build Summary: 3/5 steps succeeded (1 failed); 3/4 tests passed (1 crashed)

I’m genuinely puzzled. I couldn’t find a good way to set up the fuzz tests so I’m wondering if I messed something up on my setup. Alternatively, it could be environment related as this is arch linux and from my understanding the fuzzer backend is being compiled and linked from a system dependency.

Independently of the root cause, I’d like to thank you in advance for the help in figuring this out.
Best regards,
Henry

2

The initial error is https://codeberg.org/ziglang/zig/issues/30655. It only affects the self-hosted compiler backend (and thus debug mode on Linux), so you’re right that using any of the release modes should force the LLVM backend and work around the issue.

The last one I think is just the fuzzer’s output when it finds something. Although it’s a bit weird that it doesn’t include any references to your test code in the stack trace.

If you use the default build.zig created by 0.16 zig init and run

zig build test --fuzz --release=safe

it will log a localhost url for the web-based console to follow the fuzzing progress btw.

3

I tested out and it works, basically I only needed to remove .fuzz = optimize != .Debug from my build.zig. I guess this is a direct consequence of not having autocomplete for zig build test and my lack of complete understanding of the build system and its implications. Maybe some dedicated docs on that would benefit me for that, but nonetheless I appreciate the help!