Can I exclude a specific test from being cached, so that it’s called every time I trigger tests from the build.zig?
When running tests, the result gets cached normally, meaning as long as you don’t change the code that gets tested, the test is not called again. Normally, that makes perfect sense but when I was contributing to zBench lately, the question came up how we can integrate benchmarks as part of a tests. Here, you want to be able to call the test again, without necessarily having to change the source code that gets benchmarked (“tested”).
The alternatives I see at the moment are
build the test, i.e. make a binary that you can run again. Seems kind of ok to me, however making binary from tests? Should be a throw-away thing, not in zig-out/bin…
dump the full cache to a throw-away directory, e.g. with zig build tests --cache-dir $(mktemp -d). Seems overkill to me, and I see this becoming slow as a project grows…
Maybe you could generate a dependency with random data and make your benchmark “test” depend on that, you could even use that random data as a seed value in case you use any randomness in your tests.
Then a nice thing might be to add a build option that lets you specify the seed value and make it so that it is only randomly initialized if you don’t provide it.
That would mean that your builds are deterministic and caching per seed value, just if you don’t specify a seed value a new one gets generated every time.
I haven’t played that much with the build system, so I still don’t have that much practical experience, else I would create an example. But I think this should be possible.
const RndGen = std.rand.DefaultPrng;
test "random" {
var rnd = RndGen.init(@as(u64, @intCast(std.time.timestamp())));
const some_random_num = rnd.random().int(i32);
std.debug.print("random number is {}", .{some_random_num});
}
won’t work btw. - the caching works very well I guess it just registers “test ‘random’ passed”, so as long as you don’t change anything upstream, nothing happens if you call for the test again. So I would have to go for a more sophisticated solution.
This doesn’t quite answer your question but IMO a practical alternative is to treat benchmarks completely separate from unit tests. In this approach you wouldn’t use the test keyword at all to write benchmark cases but would rather implementing your own mini framework with benchmark specific reporting. Is there any benefit to spawning benchmark cases from within test sections?
The idea was that benchmarks test performance, and based on that relation I would expect them alongside other tests. I don’t think there is a technical benefit to write benchmarks as “tests” using Zig’s test framework (at least not at the moment).
I agree with that GitHub comment. In my experience, integrating benchmarks into “automated tests” (for example, letting a CI test outcome go red if benchmarks don’t hit their targets) has often lead to problems. Why? Well, because benchmarks tend to be flaky and thus you start getting all sorts of intermittent failures.
I guess it just registers “test ‘random’ passed”, so as long as you don’t change anything upstream, nothing happens if you call for the test again. So I would have to go for a more sophisticated solution.