This talk was a fascinating watch. Richard Hipp uses the story of SQLite’s development to exemplify the benefits of rigorous software testing. One of the methods that intrigued me was coverage testing on code conditions (“Modified Condition/Decision Coverage”), since it forces verification and justification of each decision your program makes. It can be automated using tools like kcov and gcovr. I also found this github issue about it, and it seems like kcov works well due to being compiler-agnostic.
As a relatively new Zig programmer and test author, I wondered how MC/DC coverage testing is used in-practice for Zig projects, and when it makes sense to enforce this level of scrutiny.
Some discussion questions:
Have you applied MC/DC in your Zig codebase, if so how?
If you have exercised MC/DC testing in general, what is your experience?
How might comptime conditions and lazy compilation be treated when MC/DC testing?
Some of the challenges Richard mentioned with mocking the interface to the OS are a bit easier/different in zig thanks to std.mem.Allocator and std.Io. We have testing versions of both that can (or will in the future) catch errors related to misuse.
It makes it easier to build simulators and stuff. Here’s a fun example from the standard library:
I think more extensive Io testing facilities are on the roadmap? But someone deeper in the guts or more up to date with development can confirm.
The team behind Tigerbeetle, also a database, have produced some interesting talks and blog posts about testing practices. This isn’t MC/DC, but they have a simulator which injects faults:
The zig toolchain comes with fuzz testing as well. Fuzzing can help find unexpected failures (as mentioned in the talk). It’s great to use fuzzing in combination with checkAllAllocationFailures!
Looking to the future, something like checkAllAllocationFailures but maybe for checkAllCancellations or checkAllIoFailures would help exercise error paths and improve test coverage.
Recently, I’ve started trying to use testing.expectError more when I can, both to exercise the error paths and document expectations.
A doctrine I’ve started to follow is to control external input by default, avoid using it as-is if possible, exactly so that you assert the data you depend on at the moment, and inject faults to test codepaths rather than waiting for upstream to do it for you in production.
In gamedev it’s extra annoying as you end up depending implicit behavior from compositors, GPU drivers and input device drivers:
Using a different mouse or subtle changes in mouse events may break double-click, dragging – a consumer OS update may suddenly start emitting a mouse dragged event even when standing still, or worse, breaks only in a specific high frequency polling mouse;
compositors. If wayland, compositors may behave slightly differently, if OSX, behavior may just change under your feet;
anything graphics API related;
frame ordering can change due changes in the presentation engine, not a lot to go wrong here, but still good to make sure your code doesn’t assume a frame order.
It saves a lot of time to write small wrappers and test to catch these changes or assumptions.
Controlling external input includes any code dependencies as well, asserting and testing their data, struct layouts, enum values, etc. The simpler alternative is to avoid version upgrades (i.e. the gamedev way).
A lot of mistakes are pretty obvious in retrospect, but it’s easy to assume every external input is doing their job when it works in your machine. Having asserts and simulation layers in some codebases I worked on would have saved a lot of time.
