What is fuzzing?
It’s intentionally blasting random (even invalid and malformed) data at your codebase, usually through its public interface, in an attempt to crash the code or expose unexpected bahavior, memory leaks, security vulnerabilities, or other trouble that might not be easy to predict or exercise by other means.
Historically, it seems more common to fuzz-test applications, within test harnesses, mimicking input to the application through its command-line args, socket interfaces, or other user interface controls. In zig, it is easy to fuzz test closer to the unit-test domain, and this is the flavor of fuzzing that will be covered in this document. Using zig tools to construct more sophistocated or higher-level tests should be a natural next step from there.
(Note: “fuzzing” and “fuzz testing” are synonyms, not mysteriously different things.)
How to do it in zig?
The zig toolset includes fuzz test tools, including std.testing.fuzz and std.testing.Smith as well as zig build support for running the onslaught.
One approach is to write your app/lib code and (conventional) test code, as usual, then run and tune your test code to achieve good coverage, and then, finally, author your fuzz test code. It is not convenient (to say the least) to instrument your core code or your test scaffolding for fuzzing, so it’s best to plan for that and keep your fuzz scaffold design to a minimum; it’s nice if you can repurpose some of your general testing code for fuzzing. More details and demonstration of this later.
Who is “smith”?
It’s not the name of the guy who came up with it. Think of a “smithy” - a blacksmith’s workshop or forge. “smith” refers to the test data generator. Fuzzing is about sending randomized data, so the smith’s job is to serve meaningful random data, in the types your fuzz code requests, from a stream of random bytes according to smith functions you call to constrain and type that randomness. For instance,
smith.valueRangeAtMost(0, 100);
will return a random value between 0 and 100 (but it will do so in a way appropriate to fuzz testing - you do not want to use random.intRangeLessThan(), as you won’t get the intelligent coverage you want and your implementation will be slower).
Basics
If you wrote a reverse() function and then this test…
test "reverse" {
var v1 = "hello".*;
try std.testing.expectEqualStrings(reverse(&v1), "olleh");
var v2 = "godbye".*;
try std.testing.expectEqualStrings(reverse(&v2), "eybdog");
}
…you may not be persuaded to fuzz-test this, since:
- this is on the simple, deterministic side,
- it doesn’t involve allocations (that might be missing deallocations in corner-cases)
- it doesn’t involve parallel processing (that might be prone to deadlocks, starvation, access control problems, etc.), and
- it doesn’t have an obvious security-vulnerability profile (it may not even be exposed to user input according to your application design)
(Want to see the poor-man's implementation of reverse()...?)
pub fn reverse(str: []u8) []u8 {
var c:u8 = 0;
for (0..str.len/2) |i| {
c = str[str.len - i - 1];
str[str.len - i - 1] = str[i];
str[i] = c;
}
return str;
}
Imagine that a little mistake here could have resulted in only even- or odd-length strings reversing correctly; conventional testing might not have exercised the error-prone case, but oracle-based fuzz testing, treated below, with lots of variably-lengthed strings, would expose the flaw.
But let’s try anyway; it’s overly simple for demonstration.
test "reverse2" {
const t = struct {
var data: [32]u8 = undefined;
fn fuzz_reversal(_: void, smith: *std.testing.Smith) !void {
const len = smith.slice(&data); // len will be between 0 and 32
_ = reverse(data[0..len]);
}
};
try std.testing.fuzz({}, t.fuzz_reversal, .{});
}
Then, we run the fuzzer:
zig build test -Doptimize=ReleaseSafe --summary line --fuzz=10K
That’s it! Notice that, unlike normal unit testing, comparing outputs to expectations is not required (e.g., with the expectEqualStrings() used above). Rather, crashing or causing memory leaks or thread deadlocks is a useful “output”. It’s pretty easy to determine by inspection that this contrived example appears safe and will never “crash”. If we changed the function signature of reverse() to return ![]u8, instead, and implemented it to return errors if the input buffer didn’t meet certain conditions (e.g., lowercase ASCII letters only), then we’d want to silently accept and discard those errors, rather than propagating them beyond our fuzz_reversal() function because they represent proper handling of that erroneous state. However, if we allocated memory within reverse() and propagated an OutOfMemory error, it would be appropriate to:
_ = try reverse(gpa, data[0..len]);
… this would propagate the OutOfMemory error from the fuzz_reversal() method, called by the fuzzer, and result in halting notification in the terminal.
Or, perhaps you don’t detect any leaks using std.testing.allocator, using your unit tests, but you want to make sure there aren’t corner cases where leaking does occur. You could use std.testing.allocator while fuzzing, for this purpose, but note that the performance will take a real dive, of course.
zig build test ... --fuzz
The line of code…
try std.testing.fuzz({}, t.fuzz_reversal, .{});
…is the standard call to start the testing. The first arg is for
FuzzInputOptions, not currently used, and the last arg is for a context struct you can define and provide for your fuzz function (first arg (_: void) in our fuzz_reversal).
It’s normal to run the fuzzer as:
zig build test -Doptimize=ReleaseSafe --summary line --fuzz
You want ReleaseSafe in order to test your optimized code, and in order to run the test efficiently and maximize your coverage and penetration. In the above test, we set --fuzz=10k, limiting the number of iterations of calls to our fuzz_reversal() function to 10,000. Otherwise, the fuzzer runs until the cows come home, which is often what you want. If you’re instrumenting code, to make sure you’ve got it scaffolded aright, you might use --fuzz=10 or some other low limit, so you don’t spill your terminal.
Note that, by default, values are accumulated across multiple runs. After several limited runs, output might look like:
======= FUZZING REPORT =======
Step: run test
Fuzz test: "main.test.reverse fuzz" (16729ee6e81554aa)
Runs: 1573702 -> 1574704
Unique runs: 4 -> 4
Coverage: 217/9487 -> 217/9487 (2.29%)
------------------------------
Even with the individual allocations in our test harness, a million runs like this take almost no time to run.
Weights
Assume you (think you) know that your reverse() function is much more likely to be called with strings between 8 and 40 characters long (but you still want to test lengths below and above). If you want to commit more resources to exercising that 8-40 range, you can assign weights. Here we’ll use the baselineWeights for the byte values themselves, but we’ll tip the scale for the lengths of the strings generated by the smith:
const Weight = std.Build.abi.fuzz.Weight;
const Smith = std.testing.Smith;
test "reverse3" {
const t = struct {
var data: [100]u8 = undefined; // tested lengths could be up to 100
const len_weights: []const Weight = &.{
.rangeAtMost(u8, 0, 100, 1),
.rangeAtMost(u8, 8, 40, 10), // 8-40 about 10x more likely
};
fn fuzz_reversal(_: void, smith: *Smith) !void {
const len = smith.sliceWeighted(&data, len_weights, Smith.baselineWeights(u8));
_ = reverse(data[0..len]);
}
};
try std.testing.fuzz({}, t.fuzz_reversal, .{});
}
Oracle-based fuzz testing
In many cases, you can actually test output against expectations. This is called oracle-based fuzzing. Let’s do so here by refining our unit test, then using it for a fuzz test:
const Smith = std.testing.Smith;
test "reverse4" {
const t = struct {
var data: [100]u8 = undefined;
var ref: [100]u8 = undefined;
fn reversal(str: [] u8) !void {
@memcpy(ref[0..str.len], str);
std.mem.reverse(u8, ref[0..str.len]); // reference reversal
try std.testing.expectEqualStrings(reverse(str), ref[0..str.len]);
}
fn fuzz_reversal(_: void, smith: *Smith) !void {
const len = smith.slice(&data);
try reversal(data[0..len]);
}
};
// one-off unit tests:
var v1 = "hello".*;
try t.reversal(&v1);
// fuzz test:
try std.testing.fuzz({}, t.fuzz_reversal, .{});
}
Note that separating the reversal() function from the fuzz_reversal() function allows expectation-based unit-testing code to be re-used by the fuzz test scaffolding. Then, you can instrument your unit test code as needed, but let the fuzzer hammer the code with random input afterward.
Boilerplate example
If you create a new directory and use zig init to produce a boilerplate, you’ll find an example fuzz test in src/main.zig. Be sure to use 0.16 or greater for the new smith-based fuzz support documented in this article.
More on Smith
std.testing.Smith documents Smith’s interface, but some highlights…
Eos
eos stands for “end of stream”. It’s used to terminate a “stream” of input data. The fuzzer runs mega-multiple iterations of calls on your fuzz scaffolding function, but, within the function, you can run a loop, simulating input data until this “eos”, like so:
while (!smith.eos()) switch (smith.value(enum { add_data, dup_data })) {
.add_data => {
const slice = try list.addManyAsSlice(gpa, smith.value(u4));
smith.bytes(slice);
},
.dup_data => {
// ....
This is from the example code generated by zig init. The while (!smith.eos()) loop will iterate some random number of times before breaking out. Variations are also provided; eosWeightedSimple(), for instance, allows you to designate how likely you’d like a false (“not end- of-stream”) versus a true (“yes end-of-stream”). Normally, the construct is:
while (!smith.eosWeightedSimple(9, 1)) {
for example, in which the loop is 9 times more likely to “continue” than to
break out. Notice the ! at the beginning, to follow the pattern, which reads
“while not end-of-stream”.
value()
In that boilerplate example, smith.value() is used to branch between emulating data acquisition (add_data) and running a duplication scheme (dup_data) and, within the add_data branch, it’s used again to fetch a random u4 value. Again, these are pre-generated random inputs, fetched by smith.value(); you don’t want to re-create the wheel by employing a random number generator yourself.
Bounded values
Of course, you may need values pulled from a specified range. For that, use valueRangeAtMost() and valueRangeLessThan() or the hash flavors of each of those. index() is another flavor, for random indices. Or…
Weights
valueWeighted() is another variant; this allows you to distribute the probability of getting one or another value from the range, based on the weights you assign. boolWeighted() functions like the weighted eos*() functions, introduced above, allowing you to specify the relative likelihood of a true value versus a false value.
Bytes
Perhaps one of the most common needs is for random bytes. bytes(), bytesWeighted(), and bytesWeightedWithHash() provide random bytes from the stream for this purpose.
Slices/buffers
Random slices can be generated with slice*() flavors; note that you have to provide the allocated (or stack) space for the data, of course.
Tests in various source files
As with other tests, fuzzing blocks can be in files other than main.zig or root.zig, but you either have to use addTest() in build.zig or reference the tests from within a main or root test:
const foo = @import("foo");
test {
_ = foo; // explicitly reference foo so its tests will run
}
Alternately, you can use refAllDecls(); note that the import assignments must be pub in this case:
pub const foo = @import("foo");
pub const bar = @import("bar");
test {
std.testing.refAllDecls(@This()); // implicitly reference all the pub decls
}
Running / terminal output
Note that zig build test just runs zig test under the hood, normally, but for fuzzing, zig build test --fuzz must be used, and, as shown above, the -Doptimize=ReleaseSafe arg is almost certainly wanted for fuzzing, so that you’re fuzzing your release build. When using zig test, you’ll get summary output by default, but you need the --summary line argument (or --summary all or --summary new) to see output when using zig build test.
Note that all fuzz scaffolds are run through once during a normal call to zig test or zig build test. However, fuzz input seems to prime with lots of zero-values at the start of the stream, resulting in, e.g., a while (!smith.eos()) test failing right away, and never exercising your code. You could fudge, but the best bet is to do as illustrated above, and make sure you can run as much of your test scaffolding code as possible in a normal test, so that your confidence in the fuzz run is high.
Web interface
You can also watch output at the web interface (e.g., at http://[::1]:46785/ if you pass --webui=[::1]:46785 to your zig build test)
Crash- vs. Expectation-testing (oracle-based fuzzing)
Again, Fuzzing to crash (without any std.testing.expect*() calls) is perfectly legitimate - don’t shy from fuzzing for lack of clear expectation comparisons. My first use-case was fuzz-testing a custom FFT algorithm - FFT is reversible, though (especially in my case) the reversal is lossy; nevertheless, boundaried comparisons could be constructed. I took an entropy approach, starting with simple periodic data where frequencies were fuzz-random-chosen, layering more fuzz-random-chosen frequencies, then perturbing from pure frequencies with fuzz. But I also included a pure-fuzz, pure-chaos test - sending fuzz into the FFT merely to exercise it to crash, if possible, or to leak memory. In that case, there was no expectation-comparison to be done, for, without periodicity, an FFT result can’t be reversed with any predictability.
Another pattern is to test validity at the end of a fuzz iteration. You may not be able to test accuracy (with an expectation comparison), but asserting the integrity of your state could expose problem code.
To Add
- Crash dumps and re-running failures
- Multi-threaded fuzzing
- Fuzzing infinite mode with multiple tests
- FuzzInputOptions.corpus
