What is a test in zig?

Explain
,
1
  1. Why is there a zig test subcommand? How are tests run by the build system different than tests run by the zig test command?
  2. What is the default test runner? Where is the source code for the default test runner?
  3. Are there alternative test runners / frameworks?
  4. Does the default test runner run tests in parallel? How?
  5. How does the test runner monitor its own sub-processes?
  6. How does the default test runner know when to cache a test result?
  7. How do I generate a test report? What test reports are supported?
  8. How does the test runner determine which tests should run?
  9. Why doesn’t my test run when I put it inside a struct?
  10. Can I run tests for other architectures? Using QEMU / wine?
  11. How do I skip certain tests based on the compile time information, like the target architecture?
  12. Why can’t I write a function definition within a test block? I can write a struct definition within a test block?
  13. What is the difference between a doc test and a regular test?
  14. Can I write a doc test for a symbol from another file?
  15. Why is there a test keyword?
  16. Why doesn’t a test block require an ending semi-colon (“;”)?
  17. Is a test a declaration, a function, an expression?
  18. Can I use compile time reflection on test declarations?
  19. Are tests optimized? What is the runtime safety of tests?
2
3

Quotes are from the docs.

  1. During the build, test declarations found while resolving the given Zig source file are included for the default test runner to run and report on.

  2. Probably because your struct is never used, and therefore will not be resolved, as above.

  3. Probably by using the -Dtarget flag, the same as for an executable.

  4.   test {
      if (builtin.target.os.tag == .windows) return;
      // ...
  }

  5. For the same reason you cannot write a function within a function.

  6. A doctest, like a doc comment, serves as documentation for the associated declaration, and will appear in the generated documentation for the declaration.

  7. I don’t see why you couldn’t.

  8. To distinguish tests from regular functions.

  9. Why would it?

  10. Test declarations are similar to Functions: they have a return type and a block of code. The implicit return type of test is the Error Union Type anyerror!void, and it cannot be changed. When a Zig source file is not built using the zig test tool, the test declarations are omitted from the build.

  11. I don’t think so.

  12. For zig build test, use -Doptimize. For zig test, use -O, both default to Debug.

4 Likes
4
  1. or return error.SkipZigTest for explicit skip, it will be displayed in the test statistics (assuming default test runner is used):
    if (builtin.target.os.tag == .windows) return error.SkipZigTest;
3 Likes
5

(1). zig test tests only the specified file. This can be helpful when you are writing something that isn’t reachable from your root yet, but you want to test. zig build test requires a test step and modules that it references. It will only run tests reachable from the modules you specify, like @n0s4 clarified in 8 and 9.

  1. I don’t think there are any alternative test runners (yet).
  2. As far as I have seen, it runs tests serially.

(6). I don’t think there is any caching of test results. All tests are run.

6

I’m pretty sure test results are cached, look at the --summary all output after the tests have been ran the first time:

$ zig build test --summary all
Build Summary: 5/5 steps succeeded; 4/4 tests passed
test success
├─ run test 1 passed 927us MaxRSS:1M
│  └─ zig test Debug native cached 19ms MaxRSS:33M
└─ run test 3 passed 997us MaxRSS:1M
   └─ zig test Debug native cached 19ms MaxRSS:33M
1 Like
7
  1. I don’t know for sure, but I think that test results are always cached, and are only re-run when the source changes.
8

Yes. I work mainly with microcontrollers, which means all the development is cross platform. For the large amount of code that is platform independent, I modified the standard test runner to integrate onto my microcontroller platform, which was not difficult. The simple test runner is just that. Now I can run test cases both on the microcontroller, where it really counts, and on my development platform, where it is really convenient and at any optimization level. It’s stuff like this that makes working with Zig such a pleasure.

1 Like
9

The build system invokes zig test to run tests; it passes a special flag --listen=- to enable a communication protocol between the build system and the test runner, then the test results are displayed by the build system.

It is the main function of the test runner that is compiled with the tests.
The source of the test runner is lib/compiler/test_runner.zig

See: ZTAP: TAP 14 Test Runner

No, the tests run sequentially by the test runner.
The build system invokes the build steps in parallel.

The test runner is the sub-process that runs the tests.
The build system invokes the test runner.

The test runner does not cache anything.
The build system caches the test runner. Set the flag has_side_effects in the test run build step to rerun the tests when there are no changes.

    const run_tests = b.addRunArtifact(tests);
    run_tests.has_side_effects = true;

The build system displays information about the tests according to the --summary flag (new and all are the most useful values).

@import("builtin").test_functions returns the list of test functions.
The list is populated from the tests of the root test module namespace and all other tests that are in referenced namespaces.

Because the struct namespace is not referenced.

Yes, the build system can run the test runner under various emulators:

  -fdarling,  -fno-darling     Integration with system-installed Darling to
                               execute macOS programs on Linux hosts
                               (default: no)
  -fqemu,     -fno-qemu        Integration with system-installed QEMU to execute
                               foreign-architecture programs on Linux hosts
                               (default: no)
  --glibc-runtimes [path]      Enhances QEMU integration by providing glibc built
                               for multiple foreign architectures, allowing
                               execution of non-native programs that link with glibc.
  -frosetta,  -fno-rosetta     Rely on Rosetta to execute x86_64 programs on
                               ARM64 macOS hosts. (default: no)
  -fwasmtime, -fno-wasmtime    Integration with system-installed wasmtime to
                               execute WASI binaries. (default: no)
  -fwine,     -fno-wine        Integration with system-installed Wine to execute
                               Windows programs on Linux hosts. (default: no)

Return error.SkipZigTest from your tests.

The doc tests are about defined symbols (functions or declarations) that are displayed as example code in the generated documentation.

Neither a function nor a declaration. It is a test (it is implemented as a function that accepts no arguments and returns anyerror!void).

Yes, using @import("builtin").test_functions. Note that this is not just a list of test names, you can also invoke the test function.

5 Likes
10

This is unfortunate. Some test errors are not always reproducible and can depend on some information (like current date-and-time) that is not part of any file. There should be a way to force-rerun tests when so desired.

1 Like
11

You also could set the has_side_effects to true or false based on some build-option.

3 Likes
12

IMO tests should be deterministic, and not rely on any environment conditions. I don’t know what kind of system you would test that relies on system time, but if you had to test it then simulating the time via some form of dependency injection is what I would try to do.

13

It is common to have random data generation for fuzzing and property testing.

14

But that is still deterministic and repeatable once you store the seed value for the random generator that was used in the test run.

1 Like
15

Yes, I agree. It is also best to have a mock for current time and anything else not deterministic that turns the tests to deterministic.

2 Likes
16

Unless you read from network or from dev/random

17

I would call this repeatable rather than deterministic though. It’s still a test which fails randomly, rather than always passing or always failing.

When stochastic tests like fuzzers fail, it’s good to make the failure into a deterministic test so that it fails immediately if the fix regresses.

1 Like