Data driven / Table driven testing

[nm-remarkable]

I’ve been a very big fan of data driven testing / table driven testing, however I don’t feel like this pattern is very well supported by Zig, so I wanted to know how other people approach this problem.

As far as I know the Zig standard library does not employ these testing techniques for example in ldexp.zig (Codeberg is down so copying the code here).

test ldexp {
    // subnormals
    try expect(ldexp(@as(f16, 0x1.1FFp14), -14 - 9 - 15) == math.floatTrueMin(f16));
    try expect(ldexp(@as(f32, 0x1.3FFFFFp-1), -126 - 22) == math.floatTrueMin(f32));
    try expect(ldexp(@as(f64, 0x1.7FFFFFFFFFFFFp-1), -1022 - 51) == math.floatTrueMin(f64));
    try expect(ldexp(@as(f80, 0x1.7FFFFFFFFFFFFFFEp-1), -16382 - 62) == math.floatTrueMin(f80));
    try expect(ldexp(@as(f128, 0x1.7FFFFFFFFFFFFFFFFFFFFFFFFFFFp-1), -16382 - 111) == math.floatTrueMin(f128));

    try expect(ldexp(math.floatMax(f32), -128 - 149) > 0.0);
    try expect(ldexp(math.floatMax(f32), -128 - 149 - 1) == 0.0);
}

The big advantage of data driven testing for me is that the test does not “fail fast” instead each failure will be shown individually.

Go handles this in a nice manner by using subtests aoc/go/cmd/02/02_test.go at master · nunokaeru/aoc · GitHub

In C and C++ you can use macros to create different test cases, for example in CATCH a test could look like:

TEST_CASE("int")
{
    auto [a, b] = GENERATE(table<int, int>({
        {0,0},
        {1,1},
        {2,2},
    }));
    REQUIRE(a == b);
}

I’ve found a way to (ab)use comptime to create these types of tests in Zig but I am not super happy with the approach. Here’s an example:

test "not duplicated" {
    comptime {
        const values = [_]u64{
            12,
            13,
            12312,
            41141,
            1992191921992,
        };
        for (values) |n| {
            _ = NotDuplicatedTest(n);
        }
    }
}

fn NotDuplicatedTest(
    comptime number: u64,
) type {
    return struct {
        test {
            var intBuffer: [20]u8 = undefined;
            try std.testing.expect(!duplicates(intBuffer[0..], number));
        }
    };
}

Has anyone tried to approach this topic and has a better solution for this?

[filo]

I’m doing something similar here.

It is a bit verbose, but it gets the job done; not sure if this helps you

[nm-remarkable]

That looks very similar to the Go approach but it “fails fast” right, as in If the first case fails the rest of the cases won’t be executed

[filo]

Yep, exactly.
It’s not the best approach but works good enough for my use case.
Maybe you could do something with std.Io to make them parallel via std.testing.io?

I’m just throwing ideas

[pzittlau]

This is a pattern I like to do:

const std = @import("std");
const testing = std.testing;

test {
    const as = [_]u32{ 1, 2, 5, 8, 1234 };
    const bs = [_]u32{ 1, 2, 5, 8, 1234 };

    for (as) |a| {
        for (bs) |b| {
            try oneTest(a, b);
        }
    }
}

fn oneTest(a: u32, b: u32) !void {
    errdefer std.log.err("{} {}", .{ a, b });
    try testing.expect(a * b < std.math.maxInt(u16));
}

Note the errdefer in the test function.

[filo]

Cool! I should try this

[matklad]

What I’ve learned from andrew, and what I also agree with, is that this:

// BAD
test {
    inline for(.{
        .{a1, b1, c1},
        .{a2, b2, c2},
        .{a3, b3, c3},
    }) |test_case| {
        const a, const b, const c = test_case;
        // code
    }
}

is worse than this:

// GOOD
fn check(a: A, b: B, c: C) !void {
    // code
}

test {
    try check(a1, b1, c1);
    try check(a2, b2, c2);
    try check(a3, b3, c3);
}

The function call version is shorter, and gives significantly more useful stack trace on failure, that points at the specific case (one might recall Dijkstra idea from the famous paper about adding loop counters to stack traces).

Both options are fail-fast, but I consider that to be a feature: you can order the cases from the simplest to hardest one, to create “suggested debugging order”.

With two and more-dimensional data, it makes sense to go with for loops like pzittlau suggests, to get O(N*M) tests out of O(N+M) source code.

The for loop can be improved by adding

errdefer log.err("case={}", .{test_case});

to its header, to print specific failing tests on an explicit failure, but that’s not universally useful, as, ideally, incorrect behavior is caught by std.debug.assert even before you get to try expect the wrong answer.

[nm-remarkable]

I think the non fail fast property when designing a set of tests is quite useful. Usually I find that the information obtained from how many and which X out of Y tests fail to help significantly in finding what the issue is.

I haven’t found it useful to use inline or comprime for sequential tests. Besides the “not fail fast” the examples provided also have three other benefits (at least the go and cpp examples, not entirely sure about the semantics of the zig example):

• It’s guaranteed that each test cannot affect another (unless it touches global state).
• The execution order of each test is not guaranteed
• These tests can be concurrent, it’s up to the test executor

[dennypenta]

stacktracy point really is a winner.

either using function like style or put explicit error log with a ficitonal test name

[Sze]

related:

Comptime test names

I am a new to Zig and was intrigued by the idea of what comptime could do for generating code. The first idea I wanted to try was using comptime for loops to generate a bunch of orthogonal test cases. That didn’t quite work as expected (compile error: “expected statement, found test”) until I found this forum solution: Generating Tests at Comptime - #4 by koe But, the solution does not provide a way to name the test, so this really isn’t a full solution (I want to test a subset of tests often…

[vancluever]

This is mine that I have in z2d:

/// Internal table test helper. Passes in an array of structs representing test
/// cases. "name" is the only expected field, otherwise it's entirely handled
/// by `f`.
pub fn runCases(
    name: []const u8,
    cases: anytype,
    comptime f: *const fn (case: anytype) TestingError!void,
) !void {
    for (cases) |tc| {
        f(tc) catch |err| {
            debug.print("FAIL: {s}/{s}\n", .{ name, tc.name });
            return err;
        };
    }
}

It kind of has some duplication because you have to assign your tests a name for reporting (maybe one day we’ll be able to have access to the name of a running test inside of it), but you could easily discard that part if you don’t need it.

Then from there you can just do the standard table-based tests pattern where you create a cases table and pass it in.

You could probably adjust it fairly easy too to take a filter that could be passed in as a build option.

Fail fast issues could also be easily addressed by collecting failing tests in the loop and reporting on them after.

[pzittlau]

@vancluever

maybe one day we’ll be able to have access to the name of a running test inside of it

Since a test is just a function you can inspect the debuginfo for it. I’ve done this once in 0.15.1 for a benchmarking library I did:

fn getTestName() !?[]const u8 {
    // HACK: get name of test we run in, if any
    const debug_info = try std.debug.getSelfDebugInfo();
    var it = std.debug.StackIterator.init(null, null);
    while (it.next()) |address| {
        const module = try debug_info.getModuleForAddress(address);
        const symbol_info = try module.getSymbolAtAddress(debug_info.allocator, address);
        if (mem.startsWith(u8, symbol_info.name, "test.")) {
            return symbol_info.name[5..];
        }
    }
    return null;
}

I don’t know if that still works or if any of the functions used still exist. Also of course it’s a bit hacky and doesn’t work if the binary is stripped.

[dee0xeed]

@nm-remarkable

I’ve been a very big fan of data driven testing / table driven testing

well, “data driven testing” might work, when that data format is fixed forever.
but that’s impossible, there are all those md, rst etc.

but “format” is just a “form”, mind “content”…

[filo]

Interesting!
This looks better than how I usually do this kind of tests, I need to try this.

Thanks for sharing