zig build --search-prefix $HOME --prefix $HOME test
then after the build the unit tests are all executed correctly, as expected. However, after seeing the success in the test output on the terminal, the shell command exits with an error:
I’m not sure if it’s the only root cause, but I faced the same issue and actually had tests printing to stderr (using std.debug.print for instance)
This was apparently tolerated in 0.15.2 but no longer in 0.16
This is it. Thank you. But it is even worse: I will not be able to stop my test code to print to stderr, because parts of it is in Rust and will do that when called from Zig.
This is a design error in Zig. We have the error code to determine that. Monitoring stderr if there will be text and then fail is a mistake.
The design is that you can use a custom test runner if your usecase needs something else then the default test runner.
Could the design of the default test runner be improved? maybe
Make a good case for it and it might be.
I think it is helpful to distinguish between different usecases and what kind of test runner implementation they would use; and the separate matter whether a specific implementation should be the default.
Putting every feature into the default test runner seems like it would lead towards nobody getting what they actually want, so I prefer having to use a custom one for some tests.
Thank you for the tip with the custom test runner. I didn’t know about this option. But I’m sure that failure of processes on all supported platforms is expressed by error code / exit code. To express this by output to stderr is a double. That’s why I’m calling this a design error.
While a nonzero process return code frequently indicates an error, it is far from universal. Git doesn’t always, and some system commands return a count. It is just a number whose meaning varies with the specific command/process.
I’m not unsympathetic to this argument, and for what it’s worth, I decided I’ll just live without the ability to debug-print from tests when updating one of my libraries to 0.16.
That said, I think it’s sort of disingenuous to characterize this as a niche “custom” feature. Many programmers coming from many (almost all?) languages will expect for it to be possible in Zig without a custom setup.
It’s fine for Zig to chart its own path, but the “slippery slope of support for niche features” argument simply does not land in this case.
I don’t argue for any specific behavior my argument is that there are multiple valid interpretations for how it should/could behave, depending on what kinds of tests you write and want.
So in my opinion it is best to just get on with it and customize it to your liking and if you want to you can contribute back some improvement to the default.
I think you read too much into my comment, I don’t care what the default one does, because I just created my own and got on with my project (and yes I do capture stdout and stderr of the test run and print it when I want to see it).
I am not saying what the people working on improvements (like the issue linked above) to the default should do, I think they are capable to improve it. But I am also not interested in bike shedding over what is the one true default test runner for Zig and what features it should have.
My 2 cents are either get involved and get your arguments heard, or don’t.
Zig itself changes the return value from 0 to non-zero when reading anything on stderr. It does it, because all shells treat that as failure, and that’s how it signals failure of tests to the shell. And that makes the problem. It is not possible anymore to have output to stderr without Zig returning an error value.
This is a problem, because external libraries in my test output to stderr. So I have no way to handle this correctly in the Zig test system. That renders the Zig test system unusable in this real world application.
There is a way to write parts of Zig’s test system myself. It has an interface to implement an own test runner. I expect to use this in all real-world projects; Zig’s own test implementation cannot be used in most of the real-world scenarios. The reason is, out of experience I know how common this situation is to have external dependencies, which write to stderr.
As a result, Zig’s test implementation is not usable in most projects without replacing parts of it it. This is why I call this a design error. The default implementation should be usable in all common cases, shouldn’t it?
Zig’s default test runner behavior makes another problem: how to test wanted failure?
Even in case of interpreting output to stderr as failure tests are needed for testing if this wanted behavior takes place. It must be possible to test this. That means, when output to stderr occurs (and only then) the test must succeed in this scenario, otherwise it must fail.
How can that be implemented when Zig’s test runner is interpreting itself?
The default test runner has too many features and not too less. It should not interpret anything but act on asserts only. The test code only has to define what is wanted behavior and what not. Otherwise it is not possible to test wanted failure and it is not possible to test dependencies, which do not follow the assumed conventions. Nothing should be assumed but success or failure have to be expressed explicitely, that’s the only way test implementations can work in all cases.
Have you considered rerouting stderr output for these tests to a file? That would fix the issue of the test runner seeing stderr and failing.
For the case where you expect external test to output to standard error, you could have a small wrapper test that spawns a child process, captures the stderr and check it.
I think your expectation is different that the way the test runner is designed. The default test runner is built with zig test blocks in mind. The way these work is to use the typically Zig error handling mechanisms. I think your use case is significantly distinct from this, which is why you are having friction here.