So I went over to my root.zig and added the magic lines.
test {
_ = foo;
}
and now tests actually ran.
$ zig build test --watch --summary all --error-style verbose_clear
Build Summary: 3/3 steps succeeded; 2/2 tests passed
test success
└─ run test 2 pass (2 total) 5ms MaxRSS:4M
└─ compile test debug native success 447ms MaxRSS:146M
watching 79 directories, 0 processes
I have to carefully read the output and search for 2 pass (2 total). That’s the only indication I get that tests actually ran. I don’t even know which tests ran…
Is there something like go test -v, which prints each and every test that runs? It prints the test name even if the test gets cached and doesn’t actually run.
I haven’t yet figured out when Zig will build/run something and when it’ll try to be smart and skip it. I’m constantly worrying about my code being broken.
Yes you can but the standard test runner doesn’t do this. You can find the test runner in your install location in lib/compiler/test_runner.zig. I can warmly recommend this blog post even if it’s a bit older.
Honestly I really wish the standard test runner had an option for this. I have chosen the custom test runner option, and for me, rebasing the test_runner code is one of my pain points when it comes to updating the Zig version.
Is there a consensus on what the best output format would be? TAP? Would be pretty easy to add. I’m adamant about the defaults but there’s absolutely no problem with adding a flag that does this.
I also find more verbose output in the standard test runner highly desirable. If no one else is commenting, I’ll take a stab.
I think supporting both machine readable output (zon, json?) and “plain text for humans” makes sense. The plain text mode should use ANSI escapes to color the output so that failures are red, pass is green, and skipped could be yellow. (Not sure about the yellow part, maybe other languages have an established convention for this.)
I’d find something like below pretty readable for me:
running 4 tests
test parser.basic ........ ok
test parser.utf8 ......... ok
test math.divide_by_zero . FAILED
test io.read_file ........ skipped
failures:
---- math.divide_by_zero ----
expected error
got success
test result: FAILED. 2 passed; 1 failed; 1 skipped; finished in 8.4ms
While not quite in the proposed output syntax, sharing an image of a test runner output I use in my own projects. The thing I like about colors is that it’s super fast to spot “all green” (all looks good, move on..) or “any red” (test failures, dig deeper).
The first thing I want to know after writing a new test and seeing everything pass is “did it even run?” When it says pass, is that because my test was correct or because my test wasn’t collected?
I usually just add a failure to the top of my new test and just to make sure it fails, and rerun it to make sure. Getting a list of all tests would be much more convenient, especially after a big refactor.
For the red-green color blind among us (myself included), please make sure the intensities are different, or perhaps the failures are bold while passes are not. Scanning a list of Christmas colors for the reds amongst the greens is needlessly challenging.
The first thing I want to know after writing a new test and seeing everything pass is “did it even run?” When it says pass, is that because my test was correct or because my test wasn’t collected?
I do too, which is why I support the idea that the standard test runner would get an optional verbosity option that makes it print each test that it ran, like in my proposal.
The output format I proposed (not in the screenshot) had it like so:
test parser.basic ........ ok
test parser.utf8 ......... ok
test math.divide_by_zero . FAILED
test io.read_file ........ skipped
Each line is a result of trying to run a test.
The current standard test runner’s “print nothing unless failed” makes me also second guess if my new tests even ran at all.
For the red-green color blind among us (myself included), please make sure the intensities are different, or perhaps the failures are bold while passes are not
If this gets implemented, I think it makes sense to learn how other test frameworks color pass/fail. zig build has an option to control color output with --color [auto|off|on], it makes sense to do the same with the test runner. And hopefully ok vs. FAILED is easy to distinguish with a quick eyeballing of the output.
