Simple test runner for Zig 0.16.0 wanted

[Sze]

Tests aren’t full programs, tests are essentially functions called by the test runner.
When a function fails the test succeeds can be done by using std.testing.expectError.

When the program fails the test succeeds can be done by using the build system:

• creating a run step for the program
• and check the status code via std.Build.Step.Run.expectExitCode

also take a look at this:

How to write integration tests for CLI utilities?

Let’s say I am writing a CLI utility, which, say, prints hello world: // ./main.zig const std = @import("std"); pub fn main() !void { try std.io.getStdOut().writeAll("hello world"); } Now, I want to write an integration test to run the utility and check its output, something like this: // ./tests.zig const std = @import("std"); test "it prints hello world" { const exe_path: []const u8 = get_exe_path(); const exec_result = try std.ChildProcess.exec(.{ .allocator = std.tes…

---

@fdik

Yes, testing needs the ability to have zero assumptions in the test runner and to have all tested features specified in the tests themselves. Whatever the assert asserts has to be the success, what ever the assert cannot assert has to be the failure. AFAICS this is the essence of the idea of testing, which enables to use such an infrastructure for all cases.

I don’t agree with this, this is just your opinion, my opinion is: people can and do use different test runners for their different purposes, because it allows them to adapt how test blocks/functions work to their use-case. It doesn’t enable the use-case of allowing me to just do what I want to do, which seems to be the reason to have the feature of custom test-runners to begin with.

Just because you don’t want to use test runners this way or don’t like them, doesn’t mean that you should be able to forbid me from using my custom test runner.

It seems like you have some pre-determined notion of how testing should be done and you just present it as if it was self-evident, but you can’t just say this is how I do testing, thus it shall be!

That isn’t an argument for why it should be that way, it is just you asserting your preference claiming it should be the only option.

The test-runner is what allows us to adapt tests to what the user needs, thus testing can be used for different use-cases.

The whole thing about the essence of testing isn’t convincing me.

If the argument is that we should put all code in the test, then we could just say that there is no test runner and everyone needs to run executables with std.Build.Step.Run.expectExitCode and implement everything themselves, I think that would be worse than what we have now.

If the argument is that test runners should only be allowed to do purely cosmetical changes, but no semantic changes, then that would mean that every test block now needs to include a configure section (or be even more verbose) that specifies whether output should result in failure or not, whether panics should result in success, etc.

I find it preferable to just pick a test runner that behaves the way I want it to behave and then write the tests against that test runner. For example I could write a test runner that expects every test to panic. If the test needs to specify that behavior then every test runner now needs to support that feature, thus you would actually end up with a test runner that has too many features, because it needs to support every single use case.

Seems simpler to be able to pick your test runner and what features it needs to support.
And it seems reasonable that you would have to change your tests if you change your test runner to one that has a different set of features.