Negative test filtering / excluding specific tests from the test suite

Help
1

I do test filtering with command line options in zig build.

Most of my test filters are some variation of “everything except the integration tests”.

Is there a way I can tell zig build test or zig test to exclude tests with a filter instead of including them?

3 Likes
2

+1, I find it hugely useful to separate tests on a case-by-case bases into fast tests I run locally before git push, and slow tests that are run asynchronously by the merge queue on CI. There’s Pareto principle in play here: 80% of tests finish in 20% of time.

I also found that historically it really wants to be a per-test, rather than a per test-suite thing. Some “unit” tests do combinatorial exploration that needs a lot of CPU time, some integration tests are smoke tests that finish in milliseconds.

I usually filter the two at runtime, by looking at the “RUN_SLOW_TESTS” env variable.

4 Likes
3

I think for that it would be better to just have a manual guard like this in the tests:

const config = @import("config"); // passed in from build system
const tests_to_run = config.tests_to_run;

const TestsToRun = struct {
    integration: bool,
    unit: bool,
    subsystem_a: bool,
    subsystem_b: bool,
    slow: bool,
};

test "my slow integration test" {
    if (!(tests_to_run.integration or tests_to_run.slow)) {
        return error.SkipZigTest;
    }

    try testing.expect(true);
}

and have an option in the build.zig to only run the fast tests or some other subset.

Though this can of course be improved.

6 Likes
4

Something like this is close to what I already do, except I construct a filter for the test names at build time. My main problem with it is that there’s no good way with name filtering to exclude tests in some of the codebase and not in others.

I think switching to skipping tests instead of filtering them can solve that, so I’m marking this solved.

1 Like
5

May not be relevant, but, if you’re not aware already, test filtering acts on fully qualified names:

1 Like