Generating Tests at Comptime

Help
1

Hi all, apologies in advance if this is in any way a duplicate, I’ve tried to look around but couldn’t quite find a direct solution.

I am looking to automatically generate a bunch of tests at comptime based on a test suite consisting of many, many files.

Essentially, I would like to do something like:

comptime for (fixtures) |fixture| {
    test fixture.name {
        fixture.performTest();
    }
}

Where I generate fixtures at compile time by means of essentially @embed-ing a bunch of test files.

Naturally, I can just create a single, massive test. But this prevents me from seeing which tests fail without hacking my own reporting.

Has anyone hit a similar issue before and have a creative solution?

1 Like
2

I haven’t tried it myself, but you might be able to spit out a generated source file and have a subsequent build step depend on it.
My idea would be textual code generation, though, which might be a tad flaky. Though at least you’d be able to see the files the codegen output.

I don’t know if Zig has any other code generation method available yet or not, besides textual. It can generate types. Haven’t tried it with tests. I don’t know if they’re as “first class” as compile-time type is, yet.

As someone who has a background in writing test automation full time, I see the value of very big test suites. But also consider that one of the cool things about Zig is being able to put tests RIGHT next to the code it tests, and use them as a sort of documentation. Even if you’re going to build a big test suite (please do), make sure you put a few doc tests along side the functions you’re writing as well.

3

Thanks for your reply.

Code gen is definitely a solution, although I agree that it is a tad flaky as you mention.

Based on your reply, I will probably have a read through the standard library code base to see if you can do something similar to what you can do with @Type :slight_smile: , that’s a great idea.

Responding on your point on test as documentation, to further contextualize my question, what I am writing is a comptime TOML parser that parses TOML to a Zig type (with again, @Type). To verify the library, I plan to use an existing test suite (consisting of TOML and JSON file pairs). Rather than porting each test to Zig (cumbersome + error prone, there are 500+ or so tests), I plan to automatically generate them with Zigs amazing comptime capabilities :slight_smile:.

4

As an update I found a way to do it by digging through the standard library:

comptime {
    const files = .{"hay", "hey", "hooi", "needle"};
    for(files) |file| {
        _ = struct {
            test {
                try std.testing.expect(std.mem.eql(u8, file, "needle"));
            }
        };
    }
}

This serves my purpose well :slight_smile: ! That being said if someone knows how I can give these tests names that would be even better.

1 Like