The corpus is the input, or seed IDK, of a fuzzing run.
You use it to repeat a fuzzing path so you can debug and fix issues you encounter.
Generating data from the fuzzing input is what the smith does at your direction
1 Like
ah, so I don’t touch corpus usually, but when the fuzz manages to crash the app I setup the corpus with the data from the report without --fuzz flag, right?
so it’s data used only in regular tests?
Doesn’t this above answer your question already? If not, can you make your question more specific? I don’t know more beyond that, because I haven’t had time to use the new fuzzing.
I think that means the corpus is run in both modes fuzzing and non-fuzzing.
I asked once about making Smith.constructInput() pub so that it could be used for simple fabrication of input data, as you see a lot of in the fuzz source test blocks, but I haven’t taken it further (I haven’t created an issue or PR to that effect). If I’m reading your request correctly, this is the kind of thing you’d like. However, it could be that you’re trying regular unit testing, rather than fuzz testing, if you feel that specific values are important. The other way to constrain certain data is to use weights. Look above, in this thread, for the great example of a network packet processor in which a checksum field. Obviously, if all fields were random, then the chances of a random checksum matching would be quite astronomical. In this case, the pattern of requesting a (random) bool from the Smith; if true, calculate the checksum correctly so that the “good path” in the code will be taken. If false, allow a random value, in order to make sure the “error path” behaves well. You could add weights to indicate you’d like the correct path taken 80% of the time. If your numbers “42” and “53” are so specific, you could theoretically fabricate a request for a “random” int via .rangeAtMost() to constrain it to only individual values, but I’ve never tried that.
is it possible to compile a test binary with --fuzz?
the idea is to build and run kcov on it.
with --fuzz I have to use addRunArtifact, can’t do b.addSystemCommand(&[_][]const u8{"zig-out/bin/test"});
I don’t think this is what you want, but you can use zig test -ffuzz file.zig … --ffuzz will “enable fuzz testing testing instrumentation”. But the fuzzer, when run with zig build test --fuzz, is… well, “running”. The closest thing to a capture is what’s captured in the corpus for playback later. Not sure you’ll find quite what you’re looking for.
I might see what you’re trying to do with kcov. I’m guessing this is not (directly) related to your “42 and 53” request earlier. You might be able to kcov ./out zig build test -Doptimize=ReleaseSafe --fuzz to run the fuzzer while kcov watches. Or, if you have a corpus from a previous run that you’d like to re-run while kcov watches, you can run std.testing.fuzz() with a specified .corpus set, like this. (I haven’t yet included this stuff in the doc, here, but intend to.)
kcov ./out zig build test -Doptimize=ReleaseSafe --fuzz - takes coverage of zig itself.
I would ask if it worked.
while compiling a test binary with fuzz is not straight forward
This is still new territory to me, but I installed kcov and tried my suggestion. My understanding is that zig build test --fuzz results in a three-steps: first, the maker compile, then the test compile, then the test run. Since I didn’t want kcov to instrument the compilation steps, I first ran zig build test ... --fuzz=10 to complete the compilation. Then I ran kcov ./kcov_out zig build test ... --fuzz, which immediately started the fuzzer, and aimed a browser at kcov_out/index.html. Though this ran indefinitely and kcov could inspect to its heart’s content, it didn’t work - the web browser output just showed zeros and I got, in the terminal: “kcov: warning: kcov requires binaries built with -g/-ggdb, a build-id file or GNU debug link information”. Right now, fuzzing requires -Doptimize=ReleaseSafe - this will eventually be fixed (so you’ll be able to run debug builds). Presumably, the debug symbols kcov needs will then be readily available. I’m not well-versed in adding debug symbols to ReleaseSafe builds; I tried some of the --verbose-* args, which enable compiler debug output for various purposes, but I couldn’t get the warning to go away. I think there’s hope in this approach, though, if somebody will chime in with the secret to getting those debug symbols in even under ReleaseSafe.
Sorry I don’t have time to find exact instructions. But I think you can set .strip=false on tests in build.zig. Not sure what the cli flags are, maybe -fno-strip?