Add test filters based on build system step

Help
1

Hi,
is there any way to set test filters based on a std.Build.Step.Run step? std.Build.addTest only supports passing filters as a [] const []const u8.

Context:
I want to run some microbenchmarks, which are defined in tests similar to .
To discover which tests are benchmarks I run them in debug mode with a custom test runner which reports this (very similar to how fuzz tests are detected in std). Here I also run them once to detect any safety checked IB I run into.
I then recompile my tests with -O ReleaseFast and run them with another custom test runner that handles the actual benchmarking stuff. Currently, that test runner filters (at runtime) that only benchmark tests are run.
What I want is to pass the discovered tests as a filter during recompiling. This would make the logic of my test runner simpler and also would lead to faster compile times (Only benchmarks are compiled with ReleaseFast).

2

Not the answer you’re looking for, but I tend to separate benchmarks from test for this kind of reason. Most of std.testing isn’t meant for benchmarking

2 Likes
3

I think you are over engineering this, you could just require that a test name include “benchmark”, and use that as the filter.

2 Likes
4

I probably am. I also somehow forgot that --test-filters does substring matching.
Still filtering tests based on a run step seems like something that is nice to have.

5

You could call zig test --test-filter ... directly from within the run step that determines which tests to run by using std.process.run.
From within the build script you can get the zig executable path using b.graph.zig_exe. To pass it into your program either use run.addArg(...) or go the b.addOptions() route.

6

I thought about that, but the module where my benchmarks live also imports other modules and does other build system stuff, so I dont think that works here.

1 Like
7

Yeah it’s a pickle if you got build system stuff happening. Since 0.17 is removing custom steps, and for good reason (they were never supported anyways), I can’t suggest rolling your own step. To stay compatible, the main tool you have to work with is using addOutputFileArg and addFileArg to pass data between run steps. However there’s no way for the build system to actually read that file, you just gotta pass it into another exe. Though the second exe does not need to have the same optimization mode or even the same root module as the first.