What test runner do you use?

Brainstorming
1

For a long time, I’ve been using Karl Seguin’s test runner, to get list of tests shown consistently, not depending on TTY, and also easy way of filtering which tests run, plus some custom modifications for log capture. Do people put up with the standard test runner? Is there an easier way to use external test runner, without having to copy it to every project?

1 Like
2

I am using the default test runner and I am executing the tests when a source file changes.
I am not using --watch, yet. I am using entr (Event Notify Test Runner) feeding it with the source files to watch.
My script for running tests is:

#!/bin/sh
set -eu

die() {
    echo >&2 "$@"
    exit 1
}

have() {
    command -v "$@" >/dev/null
}

if [ -d .jj ]; then
    LS='jj file list'
elif [ -d .hg ]; then
    LS='hg files'
elif [ -d .git ]; then
    LS='git ls-files'
elif have fd; then
    LS='fd --hidden .'
elif have find; then
    LS='find .'
else
    die "No list of sources."
fi

if [ -f Makefile ] && have make; then
    CMD='make test'
elif [ -f go.mod ] && have go; then
    CMD='go test -v'
elif [ -f Cargo.toml ] && have cargo; then
    CMD='cargo test --all-targets'
elif [ -f build.zig ] && have zig; then
    CMD='zig build test --summary new'
elif [ -f deno.json ] && have deno; then
    CMD='deno test'
elif [ -f package.json ] && have bun; then
    CMD='bun test'
elif [ -f package.json ] && have node; then
    CMD='node --test'
else
    die "Nothing to test."
fi

# double -c to clear the screen and the scrollback buffer
# -s to run CMD in a SHELL that keeps the PATH
$LS | entr -ccs "$CMD"
3

Can you share more about the nicer workflows? Maybe with https://asciinema.org/ ?

4

You can have the tester as a dependency in your project.
You can also expose a pub fn addTest function from the tester build.zig that handles the boilerplate and then @import the dependency to access the helper function from your project.

As an example, www.ziglang.org site uses zine (where ziglang site is your project and zine the test runner):
You project depends on the test runner:

and imports the test runner build.zig pub functions:

that are declared in the test runner build.zig:

2 Likes
5

Oh, that’s interesting, I didn’t realize you can import build.zig and use it from another project like that, but it makes total sense that you can. Yes, that should allow me to reuse the runner. Thank you.

1 Like
6

I personally need two features out of test runner:

  • Ability to (re-)run a single test I am looking at right now
  • Backlink compilation errors and runtime stack traces with the source code

In build.zig, I implement zig build -- "test name" syntax to set test filter.

VS Code doesn’t really have “re-run test at cursor” UX, so I, ahem, vibe coded good-enough-for-me UX, where I use a virtual editor as a shell, such that I can easily copy-paste the name of the test into this editor, use my “go to definition” shortcut on linkified output, and use a DWIM shortcut to reveal/focus/re-run last test.

EDIT: oh, I also have

    if (b.args != null) { // Don't cache test results if running a specific test.
        run_tests.has_side_effects = true;
    }

to make sure that, when I run a single test, it really is being run.

4 Likes
7

You can link ZTAP, instructions are in the README. Although looking at them, they use .root_source_file, which is gone now, that should be .root_module, I’ll update that soon[1].

I mostly use the builtin test runner, actually, despite having written ZTAP. I had a specific purpose in mind, and I use it there.

If your sentiment is ‘why can’t Zig’s test runner just “be normal”?’, TAP is about as normal as it gets. Usually while developing, the ephemeral ‘say nothing when tests pass’ philosophy of the built-in runner suits me well, but sometimes I want to see a report, filter it, whatever. ZTAP does things that way.

  1. Updated ↩︎

8

I’ve been burned many times in Zig by writing some code with tests, run zig build test, no error, only to find out the test was not included due to the discovery rules. That’s why I switched to the runner from http.zig.

1 Like
9

I check for that by adding a failing test in the file I want to include first.

2 Likes
10

If a test hasn’t failed, is it a test?

Ok, let’s put it differently: have you tested that it’s a test? That we can answer: NO

I am doctrinaire about this: always write the wrong answer first when writing a test. It’s the only sane option.

2 Likes