The corpus is the input, or seed IDK, of a fuzzing run.
You use it to repeat a fuzzing path so you can debug and fix issues you encounter.
Generating data from the fuzzing input is what the smith does at your direction
1 Like
ah, so I don’t touch corpus usually, but when the fuzz manages to crash the app I setup the corpus with the data from the report without --fuzz flag, right?
so it’s data used only in regular tests?
Doesn’t this above answer your question already? If not, can you make your question more specific? I don’t know more beyond that, because I haven’t had time to use the new fuzzing.
I think that means the corpus is run in both modes fuzzing and non-fuzzing.
I asked once about making Smith.constructInput() pub so that it could be used for simple fabrication of input data, as you see a lot of in the fuzz source test blocks, but I haven’t taken it further (I haven’t created an issue or PR to that effect). If I’m reading your request correctly, this is the kind of thing you’d like. However, it could be that you’re trying regular unit testing, rather than fuzz testing, if you feel that specific values are important. The other way to constrain certain data is to use weights. Look above, in this thread, for the great example of a network packet processor in which a checksum field. Obviously, if all fields were random, then the chances of a random checksum matching would be quite astronomical. In this case, the pattern of requesting a (random) bool from the Smith; if true, calculate the checksum correctly so that the “good path” in the code will be taken. If false, allow a random value, in order to make sure the “error path” behaves well. You could add weights to indicate you’d like the correct path taken 80% of the time. If your numbers “42” and “53” are so specific, you could theoretically fabricate a request for a “random” int via .rangeAtMost() to constrain it to only individual values, but I’ve never tried that.
1 Like
is it possible to compile a test binary with --fuzz?
the idea is to build and run kcov on it.
with --fuzz I have to use addRunArtifact, can’t do b.addSystemCommand(&[_][]const u8{"zig-out/bin/test"});
I don’t think this is what you want, but you can use zig test -ffuzz file.zig … --ffuzz will “enable fuzz testing testing instrumentation”. But the fuzzer, when run with zig build test --fuzz, is… well, “running”. The closest thing to a capture is what’s captured in the corpus for playback later. Not sure you’ll find quite what you’re looking for.
I might see what you’re trying to do with kcov. I’m guessing this is not (directly) related to your “42 and 53” request earlier. You might be able to kcov ./out zig build test -Doptimize=ReleaseSafe --fuzz to run the fuzzer while kcov watches. Or, if you have a corpus from a previous run that you’d like to re-run while kcov watches, you can run std.testing.fuzz() with a specified .corpus set, like this. (I haven’t yet included this stuff in the doc, here, but intend to.)
kcov ./out zig build test -Doptimize=ReleaseSafe --fuzz - takes coverage of zig itself.
I would ask if it worked.
while compiling a test binary with fuzz is not straight forward
This is still new territory to me, but I installed kcov and tried my suggestion. My understanding is that zig build test --fuzz results in a three-steps: first, the maker compile, then the test compile, then the test run. Since I didn’t want kcov to instrument the compilation steps, I first ran zig build test ... --fuzz=10 to complete the compilation. Then I ran kcov ./kcov_out zig build test ... --fuzz, which immediately started the fuzzer, and aimed a browser at kcov_out/index.html. Though this ran indefinitely and kcov could inspect to its heart’s content, it didn’t work - the web browser output just showed zeros and I got, in the terminal: “kcov: warning: kcov requires binaries built with -g/-ggdb, a build-id file or GNU debug link information”. Right now, fuzzing requires -Doptimize=ReleaseSafe - this will eventually be fixed (so you’ll be able to run debug builds). Presumably, the debug symbols kcov needs will then be readily available. I’m not well-versed in adding debug symbols to ReleaseSafe builds; I tried some of the --verbose-* args, which enable compiler debug output for various purposes, but I couldn’t get the warning to go away. I think there’s hope in this approach, though, if somebody will chime in with the secret to getting those debug symbols in even under ReleaseSafe.
Sorry I don’t have time to find exact instructions. But I think you can set .strip=false on tests in build.zig. Not sure what the cli flags are, maybe -fno-strip?
How do I interpret the coverage calculation from the web ui?
I don’t think I’ve seen it go past 1% (most often around 0.1%). So far the most sophisticated fuzz test I’ve done was similar to the “Swarm” testing approach from this post.
Should I expect it to creep up over a long enough run?
What strategies can I use to determine why the coverage isn’t going up to help decide whether I should make a more bespoke Smith or something more sophisticated?
1 Like
I think the Web UI numbers are boggus and include a lot of code that can’t be reach. For example all non fuzzing tests have a red mark next to them, even though a fuzzing test can’t call another test case. So probably all dependencies pulled by those other tests are being counted.
But the fuzzer did find a few bugs in my code. It took some efforts and I had several stages where my Smith wasn’t correctly covering the set of valid inputs and no bug was found. Then it found bug in my program simulation, not the actual program.
Using fuzz testing is hard or at least it requires training. But when the machine finds edge cases that you didn’t think of, it’s feels amazing
I have been off-grid for over a month, and won’t really get back “on” for another month, I’m afraid, but I’m checking in, and curious about whether anybody got this kcov+fuzz working, perhaps by getting debug symbols in ReleaseSafe as @Travis suggested…? I might have a window to try this myself in the next couple of days, but I probably won’t be able to update my doc with this good input (and other unfinished business) until I get back in the rhythm in October.