I have recently moved some of my C++ projects to use Zig build system. It has been great.
One thing I do miss when writing C++ code compared to Zig or Rust is the ability to colocate tests inside implementations.
My question: Is there a way to register Zig tests in C/C++ code? If an official solution is not available, any workarounds are also appreciated!
As far as I understand, Zig test functions will be collected into builtin.test_functions. It would be great if I can somehow push C/C++ test functions into this. Like I can just do
int factorial(int number) {
return number <= 1 ? number : factorial(number - 1) * number;
}
TEST_CASE("testing the factorial function") {
CHECK(factorial(1) == 1);
CHECK(factorial(2) == 2);
CHECK(factorial(3) == 6);
CHECK(factorial(10) == 3628800);
}
… and the test block / function will be registered into builtin.test_functions. Asserts and stuff might have to be implemented in C/C++ land, but that’s fine as long as there is an easy low-friction way to integrate it into Zig test runner.
builtin.test_functions is a slice of std.builtin.TestFn.
You could make a c abi compatible struct, with wrapper c abi functions and convert the list at compiler time. Then export it to c/c++.
Shouldnt it be the other way around? I want to use the Zig builtin test harness to run C/C++ tests, not building a test harness in C/C++ and run Zig tests. Sorry if that wasnt clear from my post.
Sorry, I misinterpreted what you meant, but its the same answer with the addition that you will need to make a test runner that grabs the exported c abi tests on top of zigs’.
you can mostly yoink the existing test runner and modify it.
specify a test runner in the options struct when creating a test artifact or as a flagged argument when using zig test.
Figures. I mean I only have C/C++ and no Zig code (builtin.test_fns is empty), so what you are suggesting is equivalent to just addExecutable a custom test binary in C/C++ and do the test collection manually, append to an test_fns array or something (or just use Catch2/GoogleTest for the collections and reporting).
That works, so the conclusion is “bring your own test framework” when you want to test C/C++ with Zig build system No integration into Zig compiler / addTest / test blocks.
(Do you have any different ideas on how to collect test functions? Or just like what I said, just do classic TEST_CASE that appends functions to a global array? I guess Zig test block is just syntatic sugar for that anyway)
That is nicer than writing a custom test runner just to additional cpp tests, I like that.
But I guess the take away here is cpp.run_all_tests: you need to write a test runner / test collector / test framework on C/C++ side. There is no such thing as auto collecting C/C++ tests, like how test {} does.
depending on how you interpret “autocollecting” you might argue that this is what Zig does, but don’t forget that in Zig tests need to be referenced directly or indirectly by the root file to be picked up.
If you don’t import in a test block all the files that contain tests, they won’t be run.
From that perspective Zig doesn’t do automatic collection from you.
And I guess C++ users might have the expectation that tests shouldn’t need to be registered manually, but that’s going to be one of many things they will have to get used to as they upgrade to Zig :^)
If you really want auto collection of tests, I imagine you could generate some sort of special symbol with a prefix and the string from the TEST_CASE macro (I am not entirely sure, haven’t worked with c/c++ directly in a while), something which then could be found and extracted through a linker, listing / showing you all available symbols? (Also haven’t done that, just imagining that it should be possible to create an automatic tool that way)
So I think it might be quite a bit of work, I am unsure whether something like this already exists somewhere, or if some c tool like aro or similar could be used to implement something like that without too much effort.
I guess that would be the direction I would research, if I wanted automatic test registration enough, although it would be relatively likely, that I would just explicitly register test functions somewhere, instead of finding a fancy solution.
I imagine that many c/c++ people just end up invoking some pre-existing c/c++ tool through the Zig build system, I imagine there are many different ways to write c/c++ tests, so I am unsure how Zig could even go about integrating those different possibilities more. Seems like somebody would have to care enough to create a test runner or a library that makes that sort of integration easier by understanding and searching for different kinds of test frameworks?, I imagine there are already custom solutions out there to improvise something, maybe they could be found and made more reusable. Seems like something that requires community effort (half/more-automated test finding, for example I could imagine that you could create a zig package that can be used in your build.zig to define a test step, parameterized with an option for what kind of test extractor it uses, but seems like a potentially big project).
I guess people who have worked more with c/c++ could provide more insight, I mostly have used some c libraries from Zig, my experience using c/c++ directly is quite a while back.
I guess another approach would be to try to port or package some existing c/c++ testing tool as a zig project?
Yep. I think running C++ tests will end up not even using the std.Build.addTest functionality, but instead they will build a “normal” executable and run that (without installing).
By autocollecting, I’m not saying that “it” (whether lang feature, build system, or framework) should automagically collect all the “tests” available in all compilation units / modules. Just providing a primitive for “collecting all test blocks inside this root struct and recursively for any referenced struct” is already helpful enough, as long as no test names duplicated and enumerated (which I believe is how Zig tests are included?)
(to clarify, i’m fine with enumerating the scopes to collect tests, not enumerating all the tests)
Putting it in another way, I think “collecting tests” as some sort of primitives in languages is a nice thing to have (like Rust or Zig, two of which I know). Don’t get me wrong, userland/library solution are always viable, but I see that is enough friction to make people not write any tests for their code (and hence, my wish/question if there is really some low-friction way. would be cool if adopting Zig build system also makes you write more tests somehow :D)
Create a Registry class with a static method to add a new TestFn, a static std::vector<TestFn> to store them, and a static method that iterates through that vector and calls each TestFn. If you also want to store the name of the test, use a std::map<std::string, TestFn> instead.
Define a class whose constructor takes in a TestFn and registers it into the Registry
Define a macro that declares a static instance of this class using the specific test name and function
From your Zig code, call the static Registry method that runs all the tests
The only issue you might encounter (I discuss this here) is if link-time optimization strips out all the “unused” symbols that do the auto-magic registration work for you.
Very interesting concept! To define the problem a little bit more concretely, I am operating under the assumption that ideally you would want:
int factorial(int number) {
return number <= 1 ? number : factorial(number - 1) * number;
}
TEST_CASE("testing the factorial function") {
CHECK(factorial(1) == 1);
CHECK(factorial(2) == 2);
CHECK(factorial(3) == 6);
CHECK(factorial(10) == 3628800);
}
To be “converted” under the hood in some auto generated c_tests.zig file to:
// Note this assumes a C calling convention, C++ functions are mangled and won't be as easily accessible :(
extern fn factorial(number: c_int) callconv(.c) c_int;
test "testing the factorial function" {
try std.testing.expect(factorial(1) == 1);
try std.testing.expect(factorial(2) == 2);
try std.testing.expect(factorial(3) == 6);
try std.testing.expect(factorial(10) == 3628800);
}
Where you would get a nice error on failure like:
......./src/c_tests.zig:12:5: 0x104886c in test.testing the factorial function (test)
try std.testing.expect(factorial(3) == 6);
^
To summarize, I am interpreting your question to mean you want to test C/C++ functions in Zig using Zig’s test system. This is in contrast to a different goal of I want to drive a C/C++ testing framework using Zig’s build system.
One way this could be feasible is write some code that parses your C/C++ source files looking for these particular blocks and generates the appropriate Zig testing code. Zig’s build system makes it easy to have a test step depend on this generated zig file.
However, there are some questions this method poses that need answering:
Are you allowed to have C/C++ code within one of these TEST_CASE() blocks (code other than CHECK()), and if so how should that be converted to Zig code?
How will your code generator resolve all the possible C/C++ symbols it needs for each CHECK()? It needs to hunt down all the function prototypes in order to know how to appropriately reference them from Zig code.
Do you care about back-referencing line numbers from your generated .zig file to the original source file? In my example, mapping c_tests.zig:12:5 to wherever that appears in the original C/C++ source file.
One relatively simple solution I would propose along these lines that doesn’t give you quite as much granularity but gets the job done would be the following:
TEST_CASE(test name) does some macro magic that expands into something like int ZIG_TEST_FUNC_test_name(void)
CHECK(thing_to_check) expands into something to the effect of if(!thing_to_check) return -1;
You would want to add return 0; to the end of every TEST_CASE block
Now, your code generation is much simpler, you parse through all your files, and for every TEST_CASE(test name) you generate the following Zig code:
This would enable using zig build test to show you failure points in your C/C++ code! It would just be up to you to Ctrl+F the test_name to find where in your C/C++ code the Zig testing failure corresponds to.
No integration into Zig compiler / addTest / test blocks.