Data driven / Table driven testing

Help
1

I’ve been a very big fan of data driven testing / table driven testing, however I don’t feel like this pattern is very well supported by Zig, so I wanted to know how other people approach this problem.

As far as I know the Zig standard library does not employ these testing techniques for example in ldexp.zig (Codeberg is down so copying the code here).

test ldexp {
    // subnormals
    try expect(ldexp(@as(f16, 0x1.1FFp14), -14 - 9 - 15) == math.floatTrueMin(f16));
    try expect(ldexp(@as(f32, 0x1.3FFFFFp-1), -126 - 22) == math.floatTrueMin(f32));
    try expect(ldexp(@as(f64, 0x1.7FFFFFFFFFFFFp-1), -1022 - 51) == math.floatTrueMin(f64));
    try expect(ldexp(@as(f80, 0x1.7FFFFFFFFFFFFFFEp-1), -16382 - 62) == math.floatTrueMin(f80));
    try expect(ldexp(@as(f128, 0x1.7FFFFFFFFFFFFFFFFFFFFFFFFFFFp-1), -16382 - 111) == math.floatTrueMin(f128));

    try expect(ldexp(math.floatMax(f32), -128 - 149) > 0.0);
    try expect(ldexp(math.floatMax(f32), -128 - 149 - 1) == 0.0);
}

The big advantage of data driven testing for me is that the test does not “fail fast” instead each failure will be shown individually.

Go handles this in a nice manner by using subtests aoc/go/cmd/02/02_test.go at master · nunokaeru/aoc · GitHub

In C and C++ you can use macros to create different test cases, for example in CATCH a test could look like:

TEST_CASE("int")
{
    auto [a, b] = GENERATE(table<int, int>({
        {0,0},
        {1,1},
        {2,2},
    }));
    REQUIRE(a == b);
}

I’ve found a way to (ab)use comptime to create these types of tests in Zig but I am not super happy with the approach. Here’s an example:

test "not duplicated" {
    comptime {
        const values = [_]u64{
            12,
            13,
            12312,
            41141,
            1992191921992,
        };
        for (values) |n| {
            _ = NotDuplicatedTest(n);
        }
    }
}

fn NotDuplicatedTest(
    comptime number: u64,
) type {
    return struct {
        test {
            var intBuffer: [20]u8 = undefined;
            try std.testing.expect(!duplicates(intBuffer[0..], number));
        }
    };
}

Has anyone tried to approach this topic and has a better solution for this?

2

I’m doing something similar here.

It is a bit verbose, but it gets the job done; not sure if this helps you

1 Like
3

That looks very similar to the Go approach but it “fails fast” right, as in If the first case fails the rest of the cases won’t be executed

4

Yep, exactly.
It’s not the best approach but works good enough for my use case.
Maybe you could do something with std.Io to make them parallel via std.testing.io?

I’m just throwing ideas :slight_smile:

5

This is a pattern I like to do:

const std = @import("std");
const testing = std.testing;

test {
    const as = [_]u32{ 1, 2, 5, 8, 1234 };
    const bs = [_]u32{ 1, 2, 5, 8, 1234 };

    for (as) |a| {
        for (bs) |b| {
            try oneTest(a, b);
        }
    }
}

fn oneTest(a: u32, b: u32) !void {
    errdefer std.log.err("{} {}", .{ a, b });
    try testing.expect(a * b < std.math.maxInt(u16));
}

Note the errdefer in the test function.

2 Likes
6

Cool! I should try this

7

(post deleted by author)

8

What I’ve learned from andrew, and what I also agree with, is that this:

// BAD
test {
    inline for(.{
        .{a1, b1, c1},
        .{a2, b2, c2},
        .{a3, b3, c3},
    }) |test_case| {
        const a, const b, const c = test_case;
        // code
    }
}

is worse than this:

// GOOD
fn check(a: A, b: B, c: C) !void {
    // code
}

test {
    try check(a1, b1, c1);
    try check(a2, b2, c2);
    try check(a3, b3, c3);
}

The function call version is shorter, and gives significantly more useful stack trace on failure, that points at the specific case (one might recall Dijkstra idea from the famous paper about adding loop counters to stack traces).

Both options are fail-fast, but I consider that to be a feature: you can order the cases from the simplest to hardest one, to create “suggested debugging order”.

With two and more-dimensional data, it makes sense to go with for loops like pzittlau suggests, to get O(N*M) tests out of O(N+M) source code.

The for loop can be improved by adding

errdefer log.err("case={}", .{test_case});

to its header, to print specific failing tests on an explicit failure, but that’s not universally useful, as, ideally, incorrect behavior is caught by std.debug.assert even before you get to try expect the wrong answer.

3 Likes
9

I think the non fail fast property when designing a set of tests is quite useful. Usually I find that the information obtained from how many and which X out of Y tests fail to help significantly in finding what the issue is.

I haven’t found it useful to use inline or comprime for sequential tests. Besides the “not fail fast” the examples provided also have three other benefits (at least the go and cpp examples, not entirely sure about the semantics of the zig example):

  • It’s guaranteed that each test cannot affect another (unless it touches global state).
  • The execution order of each test is not guaranteed
  • These tests can be concurrent, it’s up to the test executor
10

stacktracy point really is a winner.

either using function like style or put explicit error log with a ficitonal test name