Is it ok not using the builtin `test {}`-ing framework?

Help
1

Last time I was coding in Typescript, I get used to writing my own testing framework. In Zig, however, the test runner is already provided through the use of test {} declaration.

Below I wrote an example where I struggle to decide (futurewise) whether I should use what is given, ie. test "caseName" {...}, or I should create a dumb main(), put inside the things I want to test, and control everything myself.

Zig-way:

test "EmptyString" {
    const actual = try utils.String.lineBefore(0, "", std.testing.allocator);
    defer actual.deinit();
    try std.testing.expectEqualSlices(u8, "", actual.items);
}

test "FirstChar" {
    const actual = try utils.String.lineBefore(0, "bar", std.testing.allocator);
    defer actual.deinit();
    try std.testing.expectEqualSlices(u8, "", actual.items);
}

test "OutOfBound" {
    const actual = try utils.String.lineBefore(1000, "bar", std.testing.allocator);
    defer actual.deinit();
    try std.testing.expectEqualSlices(u8, "ba", actual.items);
}

Custom-way (let’s say, this is oversimplified version, without stats, ability to skip or focus on particular test cases, show/suppress succeeded cases, etc):

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer _ = gpa.deinit();
    const alloc = gpa.allocator();

    const cases = [_]struct { name: []const u8, at: u8, input: []const u8, expected: []const u8 }{ .{
        .name = "Empty string",
        .input = "",
        .at = 0,
        .expected = "",
    }, .{
        .name = "First char",
        .input = "bar",
        .at = 0,
        .expected = "",
    }, .{
        .name = "Out of bound",
        .input = "bar",
        .at = 1000,
        .expected = "ba",
    } };

    for (cases, 1..) |c, c_i| {
        const total = cases.len;
        const actual = try utils.String.lineBefore(c.at, c.input, alloc);
        defer actual.deinit();

        if (!std.mem.eql(u8, actual.items, c.expected)) {
            std.debug.print("đź”´ {d}/{d} failed ({s})\n", .{ c_i, total, c.name });
            std.debug.print("   ----\n", .{});
            std.debug.print("   Input:    '{s}'\n", .{c.input});
            std.debug.print("   Expected: '{s}'\n", .{c.expected});
            std.debug.print("   Actual:   '{s}'\n", .{actual.items});
            std.debug.print("   ----\n", .{});
            return error.DoesNotMatch;
        } else {
            std.debug.print("🟢 {d}/{d} succeeded ({s})\n", .{ c_i, total, c.name });
        }
    }
}

In case of the latter, I can put that loop inside the test declaration as well but then, it feels like I have to fight the default test runner’s output behaviour. Also, I looked at the test_runner.zig and of course hardly was able to understand something but I saw that it does additional checks like leaks in allocation when using std.testing.allocator. However, under the hood it simply uses the gpa, which panics anyway in the debug build mode when I exit program without deallocating things properly.

Besides that struggle, I have some thoughts along the way. For example, is it like discouraged not using default test {} suite? I haven’t yet reached the Zig build system but I saw the ability to add “addTest” step so I’m kind of worrying if I keep doing the dull “main()” way, I won’t be able to integrate it into the Zig’s building framework. Finally and more straightforwardly, if you’re a noob (like me), maybe it is better to stay safe within that “test {}” magic so that you won’t miss something you don’t understand yet.

2

See what you think of this:

fn testLineBefore(input: []const u8, at: u8, expected: []const u8) !void {
    // side note: typically the allocator parameter is first
    const actual = try utils.String.lineBefore(at, input, std.testing.allocator);
    defer actual.deinit();
    try std.testing.expectEqualSlices(u8, expected, actual.items);
}

test "lineBefore" {
    try testLineBefore("", 0, "");
    try testLineBefore("bar", 0, "");
    try testLineBefore("bar", 1000, "ba");
}

If something within a testLineBefore call fails, you’ll get a stack trace pointing to exactly which call in the "lineBefore" test failed.

5 Likes
3

I think that pattern looks amazing! :)) Thank you a lot! The questions I described above, however, are still spinning in my head and I’d love to hear some advices.

1 Like
4

Have you tried expectEqualStrings instead of expectEqualSlices? When an inequality occurs, it produces a much more detailed output of exactly what happened. Similar to your custom output.

5

Zig’s testing functionality is one of the best I’ve seen in my long career as a developer. The ability to write test cases directly into the source files without the need for an additional file is great. So instead of creating your own test files, try to better understand the capabilities of Zig test.

3 Likes