Negative test filtering / excluding specific tests from the test suite

Help
1

I do test filtering with command line options in zig build.

Most of my test filters are some variation of “everything except the integration tests”.

Is there a way I can tell zig build test or zig test to exclude tests with a filter instead of including them?

3 Likes
2

+1, I find it hugely useful to separate tests on a case-by-case bases into fast tests I run locally before git push, and slow tests that are run asynchronously by the merge queue on CI. There’s Pareto principle in play here: 80% of tests finish in 20% of time.

I also found that historically it really wants to be a per-test, rather than a per test-suite thing. Some “unit” tests do combinatorial exploration that needs a lot of CPU time, some integration tests are smoke tests that finish in milliseconds.

I usually filter the two at runtime, by looking at the “RUN_SLOW_TESTS” env variable.

3 Likes
3

I think for that it would be better to just have a manual guard like this in the tests:

const config = @import("config"); // passed in from build system
const tests_to_run = config.tests_to_run;

const TestsToRun = struct {
    integration: bool,
    unit: bool,
    subsystem_a: bool,
    subsystem_b: bool,
    slow: bool,
};

test "my slow integration test" {
    if (!(tests_to_run.integration or tests_to_run.slow)) {
        return error.SkipZigTest;
    }

    try testing.expect(true);
}

and have an option in the build.zig to only run the fast tests or some other subset.

Though this can of course be improved.

3 Likes