0.15.1 Fuzzing

Help
,
1

Hello,
I made a little Tetris game and found a bug that I think is near impossible to reproduce consistently with human input and thought I might try to use fuzzing and then just throw around assertions until I find it.
The issue is that I cannot manage to get fuzzing working in 0.15.1

My code is

test {
    const f = struct {
        fn f(context: void, input: []const u8) anyerror!void {
            _ = context;
            _ = input;
        }
    }.f;
    try std.testing.fuzz({}, f, .{});
}

and I already get an error:

➜  tetris git:(main) ✗ zig build test --fuzz
info(web_server): web interface listening at http://[::1]:35703/
info(web_server): hint: pass '--webui=[::1]:35703' to use the same port next time
thread 6501 panic: start index 1 is larger than end index 0
/home/markus/.local/share/zigup/0.15.1/files/lib/std/Build/Fuzz.zig:429:17: 0x1476e83 in addEntryPoint (std.zig)
        for (pcs[1..], 1..) |elem_addr, i| {
                ^
/home/markus/.local/share/zigup/0.15.1/files/lib/std/Build/Fuzz.zig:314:56: 0x13d3ea5 in coverageRun (std.zig)
            .entry_point => |entry_point| addEntryPoint(fuzz, entry_point.coverage_id, entry_point.addr) catch |err| switch (err) {
                                                       ^
/home/markus/.local/share/zigup/0.15.1/files/lib/std/Thread.zig:510:13: 0x1347860 in callFn__anon_79526 (std.zig)
            @call(.auto, f, args);
            ^
/home/markus/.local/share/zigup/0.15.1/files/lib/std/Thread.zig:1382:30: 0x12bd8b8 in entryFn (std.zig)
                return callFn(f, self.fn_args);
                             ^
/home/markus/.local/share/zigup/0.15.1/files/lib/std/os/linux/x86_64.zig:119:5: 0x1245d65 in clone (std.zig)
    asm volatile (
    ^

How do I get this running again? The exact same code worked in 0.14.1 and I saw no changes in the release notes.

2

Fuzzing is very unstable, it was more of a proof of concept implementation.

Eventually it will be fixed, it’s not a priority for the next release, so most likely a couple of releases away.

ofc someone might fix it in the meantime, but it won’t be guaranteed to not break again.

1 Like
3

Thanks for the info!

I think Ill just make some makeshift fuzzing to try and find the bug that way.

4

There are existing tools which can be made to work with zig, though it might be hard.

doesn’t support 0.15 but might be easy to update GitHub - kristoff-it/zig-afl-kit: Convenience functions for easy integration with AFL++ for both Zig and C/C++ programmers!

5

Hm idk its not a big thing and I think it will be simpler to just randomly select function calls and track the last five that happened or something manually

6

fuzzing is about the input to your program/functions not about calling different functions randomly.

Fuzzing is to try to test every branch of your program, tools typically use a genetic algorithm instead of pure randomness and track the execution of your code to avoid testing already tested branches and focus on untested branches.

Though, pure randomness is better than not fuzzing at all.

1 Like
7

Yea I have never really dealt with fuzzing, but for just finding a weird bug in a tetris clone I think that just calling random functions may work.

8

I suppose that fuzzing could be about calling different functions randomly if that corresponds to valid system inputs!

9

You can work around this for now by not using the self-hosted backend, but LLVM instead.