Custom Test Runners and Public Struct Decls

I’m currently venturing into exotic testing territory with a firmware project I’m working on. Generally speaking, I have a board I can communicate with from my laptop via a USB to serial adapter.

A simplified view of my setup is something like:

pub const serial_adapter_namespace = struct {
    pub fn instance() *SerialAdapter {
        if (_inst) |*v| return v else std.debug.panic("Trying to use SerialAdapter instance without calling init() first", .{});
    }

    // The use of an optional type here w/ accessor function avoids initialization order footguns
    var _inst: ?SerialAdapter = null;

    pub fn init() void {
        _inst = someInitializationFunction();
    }
};

Where someInitializationFunction basically just configures the serial port how I need it to be.

I’ve set up a very simple custom test runner that initializes my serial port before calling any of the test functions:

const std = @import("std");
const builtin = @import("builtin");
const serial_adapter = @import("serial_adapter.zig").serial_adapter_namespace;

pub const std_options: std.Options = .{
    .log_level = .info,
};

pub fn main() !void {
    serial_adapter.init();
    const out = std.io.getStdOut().writer();

    for (builtin.test_functions) |t| {
        const start = std.time.milliTimestamp();
        const result = t.func();
        const elapsed = std.time.milliTimestamp() - start;

        const name = extractName(t);
        if (result) |_| {
            try std.fmt.format(out, "{s} passed - ({d}ms)\n", .{ name, elapsed });
        } else |err| {
            try std.fmt.format(out, "{s} failed - {}\n", .{ t.name, err });
            std.process.exit(1);
        }
    }
}

fn extractName(t: std.builtin.TestFn) []const u8 {
    const marker = std.mem.lastIndexOf(u8, t.name, ".test.") orelse return t.name;
    return t.name[marker + 6 ..];
}

And an additional file containing the actual tests:

test "firstTest" {
    // Shouldn't be necessary?
    // serial_adapter.init();
    try std.testing.expect(try serial_adapter.instance().doSomething());
}

test "another" {
    // If the first test calls this, then it isn't necessary
    // serial_adapter.init();
    try std.testing.expect(try serial_adapter.instance().moduleCheck());
}

Running this test I hit my panic statement that checks if the serial port variable has been initialized:

if (_inst) |*v| return v else std.debug.panic("Trying to use SerialAdapter instance without calling init() first", .{});

Thinking about this more, I have a feeling my test runner is an executable compiled separately from the tests themselves, and thus doesn’t share a global scope, is this correct? Jamming the redundant initialization call into the first test that runs initializes the variable for the remainder of the tests, but I’d like to understand exactly what’s going on under the hood a little more to figure out how to do this cleanly.

Reading this post on a custom test runner I can start to understand their somewhat hacky solution for creating an “intialization” and “deinitialization” that runs before/after all tests based on a naming scheme.

Is there a better way to do what I’m after? Having a setup function that runs before all my tests will become even more critical when I ideally eventually run Zig’s test runner on target rather than on my laptop.

No, the test runner and the tests are in the same executable.
There is no global scope in zig. You may think the root namespace as global scope but it is actually a struct.
Your singleton code for the serial adapter looks good to me.

Are you sure that the correct main function is called?
You can test that by sending something, in your main, immediately after initializing the serial adapter. If the expected main is not called, check your runtime code that calls the main function (your start or _start symbol that is called after reset/on startup).

2 Likes

I’ll assume you are trying to run your tests on your target environment which is compiled as .freestanding. If that is the case, then I’ll also assume you have been able to build and run some trivial application that uses your serial communications. This is where the difficulties come in. There is no standard out or standard error on .freestanding compiles. I discussed this in a recent post for a Cortex-M4 target. My solution was to create a generic writer that uses my serial comm device, fork the standard library testing.zig and remove its standard error dependencies in favor of my own serial device writer, and combine everything with a custom test runner. At that point it builds like any other application and you can use options on zig test to place the binary in a convenient place for downloading. If you are interested, I can share whatever you might find useful.

1 Like

Very weird, I’ll come up with a minimum reproducible executable. I made a logging call in my custom test runner’s main that showed up, so I’m fairly certain the “correct” main is being called. I’ll post back when I have something others can replicate since the behavior I’m seeing has me scratching my head!

Not yet! The behavior I’m describing is happening when compiled for/running on my Linux laptop. This will be super useful though as that’s definitely a goal of mine. I would redirect any stdout/stderr to RTT when running on device. Interesting that testing.zig has a hard dependency on a POSIX notion of stdout, being able to supply a custom writer for stdout/stderr might be a nice feature request!

Alright, came up with a very simple example that shows the odd behavior:

build.zig

const std = @import("std");

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        .name = "sandbox",
        .root_source_file = b.path("src/main.zig"),
        .target = target,
        .optimize = optimize,
    });

    b.installArtifact(exe);

    const run_cmd = b.addRunArtifact(exe);
    run_cmd.step.dependOn(b.getInstallStep());
    if (b.args) |args| {
        run_cmd.addArgs(args);
    }
    const run_step = b.step("run", "Run the app");
    run_step.dependOn(&run_cmd.step);

    const test_step = b.step("test", "Run tests");

    const serial_tests = b.addTest(.{
        .root_source_file = b.path("src/serial_tests.zig"),
        .target = target,
        .optimize = optimize,
        .test_runner = b.path("src/serial_test_runner.zig"),
        .single_threaded = true,
    });

    const run_serial_tests = b.addRunArtifact(serial_tests);
    run_serial_tests.step.dependOn(b.getInstallStep());
    if (b.args) |args| {
        run_serial_tests.addArgs(args);
    }
    test_step.dependOn(&run_serial_tests.step);

    const exe_unit_tests = b.addTest(.{
        .root_source_file = b.path("src/main.zig"),
        .target = target,
        .optimize = optimize,
    });
    const run_exe_unit_tests = b.addRunArtifact(exe_unit_tests);
    test_step.dependOn(&run_exe_unit_tests.step);
}

platform.zig

const builtin = @import("builtin");
const std = @import("std");

pub const serial = struct {
    pub const SerialDevice = struct { a: usize };
    pub fn instance() *SerialDevice {
        if (_inst) |*v| return v else std.debug.panic("Trying to use SerialDevice instance without calling init() first", .{});
    }

    // The use of an optional type here w/ accessor function avoids initialization order footguns
    var _inst: ?SerialDevice = null;
};

pub fn init() !void {
    serial._inst = serial.SerialDevice{ .a = 42 };
}

serial_test_runner.zig

const std = @import("std");
const builtin = @import("builtin");
const platform = @import("platform.zig");
const serial = platform.serial;

pub const std_options: std.Options = .{
    .log_level = .info,
};

pub fn main() !void {
    const out = std.io.getStdOut().writer();

    try std.fmt.format(out, "About to call init()\n", .{});
    try platform.init();
    try std.fmt.format(out, "Called init(), instance() field 'a' is 42 as expected: {d}\n", .{serial.instance().a});

    for (builtin.test_functions) |t| {
        const start = std.time.milliTimestamp();
        const result = t.func();
        const elapsed = std.time.milliTimestamp() - start;

        const name = extractName(t);
        if (result) |_| {
            try std.fmt.format(out, "{s} passed - ({d}ms)\n", .{ name, elapsed });
        } else |err| {
            try std.fmt.format(out, "{s} failed - {}\n", .{ t.name, err });
            std.process.exit(1);
        }
    }
}

fn extractName(t: std.builtin.TestFn) []const u8 {
    const marker = std.mem.lastIndexOf(u8, t.name, ".test.") orelse return t.name;
    return t.name[marker + 6 ..];
}

serial_tests.zig

const std = @import("std");

const platform = @import("platform.zig");
const serial = platform.serial;

test "firstTest" {
    try std.testing.expect(serial.instance().a == 42);
}

test "secondTest" {
    try std.testing.expect(serial.instance().a == 42);
}

For completeness, the empty main.zig:

const std = @import("std");

pub fn main() !void {}

And finally, the output of running zig build test:

About to call init()
Called init(), instance() field 'a' is 42 as expected: 42
panic: Trying to use SerialDevice instance without calling init() first
/home/hayden/Documents/zig/sandbox/src/platform.zig:7:54: 0x103508a in instance (test)
        if (_inst) |*v| return v else std.debug.panic("Trying to use SerialDevice instance without calling init() first", .{});
                                                     ^
/home/hayden/Documents/zig/sandbox/src/serial_tests.zig:7:43: 0x1035020 in test.firstTest (test)
    try std.testing.expect(serial.instance().a == 42);
                                          ^
/home/hayden/Documents/zig/sandbox/src/serial_test_runner.zig:19:30: 0x1036f87 in main (test)
        const result = t.func();
                             ^
/home/hayden/.zvm/0.13.0/lib/std/start.zig:524:37: 0x10355b1 in posixCallMainAndExit (test)
            const result = root.main() catch |err| {
                                    ^
/home/hayden/.zvm/0.13.0/lib/std/start.zig:266:5: 0x10350f1 in _start (test)
    asm volatile (switch (native_arch) {
    ^

And then, to cap off the weirdness, modifying firstTest like so behaves correctly:

test "firstTest" {
    try platform.init();
    try std.testing.expect(serial.instance().a == 42);
}

Output of zig build test:

About to call init()
Called init(), instance() field 'a' is 42 as expected: 42
firstTest passed - (0ms)
secondTest passed - (0ms)

Yes, you are right. There are two platform.serial.
It is easy to workaround:
Export platform from the test runner and then use it from the test cases.

serial_test_runner.zig

pub const platform = @import("platform.zig");
const serial = platform.serial;

serial_tests.zig

const platform = @import("root").platform;
const serial = platform.serial;

Interesting! Is this specific to how Zig handles compiling tests or is it just because I import the same struct in two different files that aren’t “linked” in any explicit way (one importing the other)? Coming from C/C++ it’s hard to get the idea of “translation units” out of my head so this behavior is somewhat surprising!

For instance, something like this still matches my expectations:

const std = @import("std");

const namespace1 = struct {
    pub const platform = @import("platform.zig");
};

const namespace2 = struct {
    pub const platform = @import("platform.zig");
};

pub fn main() !void {
    try namespace1.platform.init();
    std.debug.print("namespace1: {d}\n", .{namespace1.platform.serial.instance().a});
    std.debug.print("namespace2: {d}\n", .{namespace2.platform.serial.instance().a});
}

Produces:

namespace1: 42
namespace2: 42