Simple test runner for Zig 0.16.0 wanted

[fdik]

Unfortunately, I am not able to find a simple test runner for Zig, which is usable with Zig 0.16.0. I am not aware how to write one, because I cannot find documentation about the topic, which refers to Zig 0.16.0. Is anyone aware of a simple test runner, which does not return an error when the tests do not fail but still send data to stderr? The test runner needs no other feature, it would be enough to have one, which fails when one of the tests fail, and which does not when no tests fail. (Complex solutions with hundreds lines of source for such a simple task are not in focus here.)

In case there is documentation about how to write a test runner in Zig 0.16.0 a link to this documentation would be preferred.

[lalinsky]

This is not as simple as you wanted, but I think you can simplify it. It’s a modification of the test runner from http.zig that runs on Zig 0.16:

test_runner.zig

Remove log capture, filtering, verbosity modes and you have what you wanted.

[fdik]

Wow, this is huge. I was more thinking about a 10 liner or something like that.

[dimdin]

See the source of the default test runner. It is distributed with zig in path lib/compiler/test_runner.zig.
Locate the fn mainSimple() in the test runner; it is the simplest runner.
@import("builtin").test_functions is used for tests reflection (tests are functions without arguments that return !void).

[Sze]

I haven’t upgraded my test runner to 0.16 yet, but I think for simple runners not much should have changed.

I found the resources I mentioned here helpful:

Basic std.Io.File.stdout (0.16)

For tests there is another option, write your own test runner which handles print to stdout and/or stderr differently (however you like it to be handled). For example I wrote a custom test runner for testing raylib examples where I run the test in a subprocess and capture stdout and stderr, but I only show that output if the test signaled via non-zero exit code that it should be considered a test failure. (Because treating output as an error isn’t useful in every testing scenario and by using a…

[theo]

I have a shorter, simpler runner (still some dozens of lines) that might be useful for reference.

It mostly follows the approach that you said you wanted – i.e., the test runner exits with 0 if no test failed or leaked, regardless of what they printed to stderr. The other thing I wanted, which you probably don’t, is timing data. But you could just remove that part.

test_runner.zig

I should also apologize to @Sze for getting a bit salty in an earlier thread about this. Setting up a custom runner was not as much of a hassle as I feared. I still find the behavior of the built-in runner quite puzzling, but it is what it is.

[andrewrk]

what’s y’all gripe with the builtin test runner? I forgot

[fdik]

It has too many features. It tests if there is output to stderr and values this as failure. It is quite common for libraries to output on stderr when compiled in debug mode. So it cannot be used in many real world scenarios, because it is common to depend on external libs.

It can also not be used for testing own software, because testing failure is not possible. Even when own software would only write to stderr in case of failure the test runner cannot be used, because testing for correct failure requires the test to succeed if the failure state was correctly arrived.

I cannot use the default test runner in any project for these reasons. I even cannot imagine a project where I could use it.

[dimdin]

From your response, I am understanding that you simply need a testing configuration that can ignore output on stderr—similar to the std.testing.log_level that controls the log lever that is considered a testing error. Is that correct?

[Sze]

@fdik

It can also not be used for testing own software, because testing failure is not possible. Even when own software would only write to stderr in case of failure the test runner cannot be used, because testing for correct failure requires the test to succeed if the failure state was correctly arrived.

Can you rephrase this? I don’t understand what you are trying to say with this.

[fdik]

I’m not sure how this is related to a log level here, but I will check. Thank you for the hint!

Yes, testing needs the ability to have zero assumptions in the test runner and to have all tested features specified in the tests themselves. Whatever the assert asserts has to be the success, what ever the assert cannot assert has to be the failure. AFAICS this is the essence of the idea of testing, which enables to use such an infrastructure for all cases.

[fdik]

Testing can mean to test if there is the expected output on stderr. Testing can mean to test if a program fails as expected on a certain type of input: “when the program fails the test succeeds”.

[Sze]

Tests aren’t full programs, tests are essentially functions called by the test runner.
When a function fails the test succeeds can be done by using std.testing.expectError.

When the program fails the test succeeds can be done by using the build system:

• creating a run step for the program
• and check the status code via std.Build.Step.Run.expectExitCode

also take a look at this:

How to write integration tests for CLI utilities?

Let’s say I am writing a CLI utility, which, say, prints hello world: // ./main.zig const std = @import("std"); pub fn main() !void { try std.io.getStdOut().writeAll("hello world"); } Now, I want to write an integration test to run the utility and check its output, something like this: // ./tests.zig const std = @import("std"); test "it prints hello world" { const exe_path: []const u8 = get_exe_path(); const exec_result = try std.ChildProcess.exec(.{ .allocator = std.tes…

---

@fdik

Yes, testing needs the ability to have zero assumptions in the test runner and to have all tested features specified in the tests themselves. Whatever the assert asserts has to be the success, what ever the assert cannot assert has to be the failure. AFAICS this is the essence of the idea of testing, which enables to use such an infrastructure for all cases.

I don’t agree with this, this is just your opinion, my opinion is: people can and do use different test runners for their different purposes, because it allows them to adapt how test blocks/functions work to their use-case. It doesn’t enable the use-case of allowing me to just do what I want to do, which seems to be the reason to have the feature of custom test-runners to begin with.

Just because you don’t want to use test runners this way or don’t like them, doesn’t mean that you should be able to forbid me from using my custom test runner.

It seems like you have some pre-determined notion of how testing should be done and you just present it as if it was self-evident, but you can’t just say this is how I do testing, thus it shall be!

That isn’t an argument for why it should be that way, it is just you asserting your preference claiming it should be the only option.

The test-runner is what allows us to adapt tests to what the user needs, thus testing can be used for different use-cases.

The whole thing about the essence of testing isn’t convincing me.

If the argument is that we should put all code in the test, then we could just say that there is no test runner and everyone needs to run executables with std.Build.Step.Run.expectExitCode and implement everything themselves, I think that would be worse than what we have now.

If the argument is that test runners should only be allowed to do purely cosmetical changes, but no semantic changes, then that would mean that every test block now needs to include a configure section (or be even more verbose) that specifies whether output should result in failure or not, whether panics should result in success, etc.

I find it preferable to just pick a test runner that behaves the way I want it to behave and then write the tests against that test runner. For example I could write a test runner that expects every test to panic. If the test needs to specify that behavior then every test runner now needs to support that feature, thus you would actually end up with a test runner that has too many features, because it needs to support every single use case.

Seems simpler to be able to pick your test runner and what features it needs to support.
And it seems reasonable that you would have to change your tests if you change your test runner to one that has a different set of features.

[theo]

If I understand it correctly, the built-in test runner will fail any test that ends up calling std.log at the .err level. And this behavior does not appear to be configurable.

Treatment of the other log levels varies and is configurable. By default, logs at the .info and .debug level are suppressed, while .warn and above* are printed. If you want, you can adjust the “minimum log level to print in testing” via std.testing.log_level.

* By “above” I mean “more severe” (the way the enum of log levels is written, lower numbers correspond to higher severity).

[theo]

Unless I’m mistaken, std.testing.log_level controls only print behavior. The logic in the built-in runner that fails a test if it emits a .err-level std.log message is hardcoded.

[fdik]

Then let’s agree to disagree.

I do not understand how you would test for reaching expected failure states. And I do not understand why a neutral test runner without assumptions should not be possible – I even think this is the easiest one to write, let’s remove any feature, which adds assumptions like the one watching stderr, let’s use assert in tests (who doesn’t), and that’s it. But of course I accept that you see this different.

[Lowell]

I believe the issue can be split into two common but meaningfully different cases:

• Testing for failure (the function must fail, but also implicitly must not use stderr to log that failure)
• Using external dependencies that log to sterr, even in the case of success

[pzittlau]

Also note that there is a very basic runner implemented that may be more like you want it: codeberg.

You would just need to change this flag to always be true.

[Sze]

@fdik

And I do not understand why a neutral test runner without assumptions should not be possible

It absolutely is possible, I just don’t think that the Zig default should be the bare minimum lowest common denominator.

@fdik

I even think this is the easiest one to write, let’s remove any feature, which adds assumptions like the one watching stderr, let’s use assert in tests (who doesn’t), and that’s it.

Here is the absolute bare minimum runner:

pub fn main() !void {
    for (@import("builtin").test_functions) |t| try t.func();
}

you add this to your test step and you can have almost no features (you do get stack trace on first error, you don’t get leak checks):

const exe_tests = b.addTest(.{
    .root_module = exe.root_module,
    .test_runner = .{
        .mode = .simple,
        .path = b.path("nofeatures_runner.zig"),
    },
});

For runners that report leaks, run all tests (instead of just up to the first failing one), print a summary, etc., you would do something more similar to what @theo has linked above.

[dimdin]

OH!, thank you for correcting me!
Yes, the error level that causes the test to fail is hard coded in test runner. std.testing.log_level is only about displaying the logs.