Fuzz Testing Demo

tensorush · August 6, 2024, 7:36am

Soo nice that I had to share it here asap!

squeek502 · August 6, 2024, 7:48am

The PR:

github.com/ziglang/zig

introduce a fuzz testing web interface

ziglang:master ← ziglang:fuzz

opened 03:19AM - 06 Aug 24 UTC

andrewrk

+3877 -1065

Makes it so that when you pass `--fuzz` to `zig build`, the build runner starts …listening on a port (configurable with `--port`) and serves a fuzzing web interface. The web interface shows source code, sharing the same Zig code from Autodocs, displaying points of interest inline as red or green dots that indicate code coverage. These points correspond to the PC addresses that LLVM selects with the sancov pass by inserting an inline 8bit counter at every control flow graph edge. The data is live-updating via the WebSocket standard. closes #20812 #352 remains open because the system is not yet integrated with unit testing and other kinds of code coverage combination. However this is obviously a step towards completing that issue. ## Demo https://github.com/user-attachments/assets/55d13596-234e-40e3-87dc-5dd0cf64aa87 ## Screenshots ![image](https://github.com/user-attachments/assets/f140cd64-d28b-40c2-a47c-211f7188fac1) ![image](https://github.com/user-attachments/assets/084a058c-f3ac-4efd-a6f1-4280f3d87051) ## Follow-Up Work * separate the build runner process from the configure process (prerequisite to #14286) * make the source hyperlinks work (currently they construct autodoc URLs) * create a UI element for navigating to any source file that is available with a quick filter box * support PIEs that have base address relocated on process start * make no `--port` arg print a hint ("use --port x to reuse this port next time") * use `.zig-cache/f` for corpus directory, keyed on fqn unit test name * support macho, coff * support more than 1 ELF module * debug info audit - a lot of the PCs have strange source locations * handle aggregation of more than one coverage_id * upgrade from u8 to usize element types for covered pc bits, don't read past end of 8bitcounters array * move the covered pc bits to after the header so it goes on the same page with the other rapidly changing memory * play a little jingle in the browser when it finds a bug! * put the source view into a scrolling box so the main UI doesn't go offscreen * show a list of files sorted by how many uncovered points of interest they have in them * unreachable code paths need to be excluded from having coverage instrumentation * include the concept of a set of modules of interest in coverage - you still want instrumentation for the purposes of fuzzing but you only want to find bugs in some code * `-OReleaseSafe` breaks everything * make it scroll to PCs that newly gain coverage * There should be a way to gain access to the interesting inputs while the fuzzing process is still ongoing. * More stats * #352 - solvable by running each unit test once after rebuilding in fuzz test mode in order to notice the coverage

mnemnion · August 6, 2024, 5:35pm

I love everything about this. Especially the visualizer, good instrumentation with a high-bandwidth means of communicating results is a killer combination.