Trouble understanding error from the fuzzer

pierrelgol · August 27, 2024, 2:53pm

Hi, so I’m currently working on a project, and the backbone of that project is an implementation of Astar. currently I’m just poking around and I’m trying to be rigorous with my testing because I hope to make a generic implementation. Anyway as I’m coding I’m using the testing framework of zig, and I’m also trying to use the new ‘–fuzz’ everything works great so far so good. But I can’t help but feel a bit confused about one error that I’m getting, either I’ve found a bug (I don’t think that’s the case), or I’m not understanding the way the fuzzer works, or I’m not understanding something in the type system. Either way I would greatly appreciate any help.

So the Test in question is this little fella :

fn fakeScores(start: []const u8, end: []const u8) f32 {
    var score: f32 = 0;

    const len = @min(start.len, end.len);
    if (len == 0) return (score);
    for (start[0..len], end[0..len]) |s, e| {
        score += @floatFromInt((s -| e));
    }
    return (score);
}

test "compare(self: *const Node, other: *const Node) Order" {
    const start = testing.fuzzInput(.{});
    const end = testing.fuzzInput(.{});
    var node_a = AstarNode([]const u8).init(start);
    node_a.h_score = fakeScores(node_a.item, end);
    node_a.update();
    var node_b = AstarNode([]const u8).init(start);
    node_b.h_score = fakeScores(node_b.item, end);
    node_b.update();
    try testing.expectApproxEqRel(fakeScores(start, end), node_a.f_score, 0.0001);
    try testing.expect(node_a.compare(&node_b) == Order.eq);
}

When running the following zig build test --fuzz --port 12345 I’m getting this error message stating that I’ve crashed with a segmentation fault.

info: web interface listening at http://127.0.0.1:44785/
test
└─ run test failure
Segmentation fault at address 0x7e2cda45b000
/home/pollivie/workspace/personnal-playground/a-star/src/astar.zig:71:28: 0x1155da1 in fakeScores (test)
    for (start[0..len], end[0..len]) |s, e| {
                           ^
/home/pollivie/workspace/personnal-playground/a-star/src/astar.zig:81:30: 0x115586e in test.compare(self: *const Node, other: *const Node) Order (test)
    node.h_score = fakeScores(node.item, end);
                             ^
/home/pollivie/zig/0.14.0-dev.1304+7d54c62c8/files/lib/compiler/test_runner.zig:155:33: 0x107f2b5 in mainServer (test)
                    test_fn.func() catch |err| switch (err) {
                                ^
/home/pollivie/zig/0.14.0-dev.1304+7d54c62c8/files/lib/compiler/test_runner.zig:55:26: 0x107dfbc in main (test)
        return mainServer() catch @panic("internal test runner failure");
                         ^
/home/pollivie/zig/0.14.0-dev.1304+7d54c62c8/files/lib/std/start.zig:605:22: 0x107d643 in posixCallMainAndExit (test)
            root.main();
                     ^
/home/pollivie/zig/0.14.0-dev.1304+7d54c62c8/files/lib/std/start.zig:250:5: 0x107d26f in _start (test)
    asm volatile (switch (native_arch) {
    ^
???:?:?: 0x3 in ??? (???)
Unwind information for `???:0x3` was not available, trace may be incomplete

The reason I’m a bit confused is because I can’t really see what I’m doing wrong, as I’m not sure what is causing it, obviously the error message is quite detailed and would be very helpful in other circumstances, but I don’t see why my for loop is leading to a segmentation fault. So I’m probably missing something and would greatly appreciate if anyone could point me in the right direction.

markus · August 27, 2024, 3:02pm

I have never tried the fuzzer myself but maybe older input becomes invalidated when asking for newer one. Try getting both the inputs as one slice and then reslice them into their respective start and end parts.

pierrelgol · August 27, 2024, 3:11pm

Thanks, it does seem to solve the problem so I was probably using the fuzzer wrong, maybe you are supposed to use only one fuzzInput() per test which would make sense, now that I think about it, one more instance of RTFM I suppose, glad I’ve found where that was coming from. I will mark you as solution, later I hope to get more insights before on the specifics if anyone knows more about the fuzzer and how it works.

Edit :

Also for the web ui, green dot means branch tested and red dot means branch not tested right ? so what is the color of an error if there is any ?

markus · August 27, 2024, 3:18pm

I think the place to check here is the zig source

I was looking for it but where the heck is the root file of the stdlib

dimdin · August 27, 2024, 3:29pm

I believe that both testing.fuzzInput calls return the same buffer.

pierrelgol · August 27, 2024, 4:14pm

Maybe not I’ve look in the repo and I think the source of my issue come from here

test_runner.zig
from this blob

extern fn fuzzer_next() FuzzerSlice;
extern fn fuzzer_init(cache_dir: FuzzerSlice) void;
extern fn fuzzer_coverage_id() u64;

pub fn fuzzInput(options: testing.FuzzInputOptions) []const u8 {
    @disableInstrumentation();
    if (crippled) return "";
    is_fuzz_test = true;
    if (builtin.fuzz) {
        if (entry_addr == 0) entry_addr = @returnAddress();
        return fuzzer_next().toSlice();
    }
    if (options.corpus.len == 0) return "";
    var prng = std.Random.DefaultPrng.init(testing.random_seed);
    const random = prng.random();
    return options.corpus[random.uintLessThan(usize, options.corpus.len)];
}

So it does seem that calling it multiple time gives a different slice. so yeah it was probably the reason why it was crashing.

export fn fuzzer_next() Fuzzer.Slice {
    return Fuzzer.Slice.fromZig(fuzzer.next() catch |err| switch (err) {
        error.OutOfMemory => @panic("out of memory"),
    });
}

dimdin · August 27, 2024, 6:21pm

Unfortunately I am right. The pointers are the same.

test.zig

const std = @import("std");

test {
    const start = std.testing.fuzzInput(.{});
    const end = std.testing.fuzzInput(.{});
    try std.testing.expect(start.ptr != end.ptr);
}

❯ zig-dev test test.zig 
1/1 test.test_0...FAIL (TestUnexpectedResult)
/home/din/zig/master/lib/std/testing.zig:546:14: 0x104073f in expect (test)
    if (!ok) return error.TestUnexpectedResult;
             ^
/home/din/test.zig:6:5: 0x10408b0 in test_0 (test)
    try std.testing.expect(start.ptr != end.ptr);
    ^
0 passed; 0 skipped; 1 failed.
1 fuzz tests found.
error: the following test command failed with exit code 1:
/home/din/.cache/zig/o/5ef5b66e41159a3a87aa749b9806c228/test --seed=0xe9faa1fa

pierrelgol · August 27, 2024, 6:28pm

Hum I don’t understand now ahah, then what was the issue ? if they are the same pointers wouldn’t that be okay to do what I initially did ? or am I missing something here
I’m confused, on top of that why did using the same slice for both did work ? if they are the same pointer shouldn’t that lead to the intended behavior ?

dimdin · August 27, 2024, 8:42pm

The following code does not crash.

const std = @import("std");

fn fakeScores(start: []const u8, end: []const u8) f32 {
    var score: f32 = 0;

    const len = @min(start.len, end.len);
    if (len == 0) return (score);
    for (start[0..len], end[0..len]) |s, e| {
        score += @floatFromInt((s -| e));
    }
    return (score);
}

test {
    const start = std.testing.fuzzInput(.{});
    const end = std.testing.fuzzInput(.{});
    _ = fakeScores(start, end);
}

The problem must be something related to using node.item instead of start in:

    var node_a = AstarNode([]const u8).init(start);
    node_a.h_score = fakeScores(node_a.item, end);

Consider that start.ptr == end.ptr, start.len may be differ from end.len, but the loop always runs for the minimum of len.

What is the code of AstarNode and AstarNode.init?

pierrelgol · August 28, 2024, 12:04am

There isn’t much honestly, which is why I find it confusing.

pub fn AstarNode(comptime T: type) type {
    return struct {
        const Node = @This();
        item: T,
        h_score: f32,
        g_score: f32,
        f_score: f32,
        parent: ?u32,

        pub fn init(item: T) Node {
            return .{
                .item = item,
                .h_score = 0,
                .g_score = 0,
                .f_score = 0,
                .parent = null,
            };
        }

        pub fn update(self: *Node) void {
            self.f_score = self.h_score + self.g_score;
        }

        pub fn compare(self: *const Node, other: *const Node) Order {
            if (self.f_score < other.f_score)
                return Order.lt
            else if (self.f_score > other.f_score)
                return Order.gt
            else
                return Order.eq;
        }

        pub fn format(
            self: @This(),
            comptime fmt: []const u8,
            options: std.fmt.FormatOptions,
            writer: anytype,
        ) !void {
            _ = fmt;
            _ = options;
            try writer.print("[item : {any}, g_score : {e}, h_score : {e}, f_score : {e}, parent : {?}] ", .{ self.item, self.g_score, self.h_score, self.f_score, self.parent });
        }
    };
}

andrewrk · August 28, 2024, 12:35am

The fuzzer is using an array list for the input buffer and resizing it. So you might sometimes get the same base pointer, sometimes not.
the API is you get 1 fuzz input per unit test run. Calling it 2x in the same test run is illegal, so that’s why you’re getting strange behavior. You can’t keep the input around for the next test run. Even if it didn’t crash, it would feed the fuzzer wrong data, making it useless.

I will change the API to make this misuse impossible, something like:

test "example" {
    try std.testing.fuzz(testOne, options);
}

fn testOne(input: []const u8) !void {
    // now it is clear you only get one input
}

By the way, I don’t think fuzzing is ready for beta testing yet.

pierrelgol · August 28, 2024, 2:11am

Thanks for the clarifications, this make sense, and yes I know that I live on the edge, but it’s also how one can hope to report issues early, so I don’t mind especially considering none of what I work on is “production grade” anyway. Thanks everyone