Using kcov with zig test

[mperillo]

Recently I tried to use a coverage tool for simple zig test and I found Code Coverage for Zig - Zig NEWS (thanks to @squeek502).

Unfortunately the example is no longer working due to regression: `zig test` on a std lib file errors with 'file exists in multiple packages' · Issue #14504 · ziglang/zig · GitHub, and testing std with filters has a annoying side effects: coverage of many files is 100% just because the test at the end of the file is evaluated.

Is there a simple way to avoid this issue?
By simple I mean that I can use a script to wrap the test command; currently I using:

zig test lib/std/std.zig --test-filter "$1" --zig-lib-dir lib \
    --test-cmd kcov --test-cmd build/coverage --test-cmd-bin

P.S: what about add a tooling tag? In future I would like to learn how to use a profiler and debugger with Zig.

Thanks.

[dimdin]

You can run all the tests and filter the coverage that is interesting to you using filters: GitHub - SimonKagstrom/kcov: Code coverage tool for compiled programs, Python and Bash which uses debugging information to collect and report data without special compilation options

Normally I am invoking kcov from build.zig:

    // tester:
    const tester_exe = b.addExecutable(.{
    ...

    // cover: tester
    const run_cover = b.addSystemCommand(&.{
        "kcov",
        "--clean",
        "--include-pattern=src/",
        b.pathJoin(&.{ b.install_path, "cover" }),
    });
    run_cover.addArtifactArg(tester_exe);
    const cover_step = b.step("cover", "Generate test coverage report");
    cover_step.dependOn(&run_cover.step);

In this example I am using an executable for fuzzing, the output is at zig-out/cover, the --clean flag erases previous coverage runs since I don’t merge the results and the flag: --include-pattern=src/ includes only my source code.

@mperillo

In future I would like to learn how to use a profiler and debugger with Zig.

It’s the same with C.
perf just works for linux and normally you can use your platform specific profiler with zig.
You can use lldb or gdb, or editors such as visual studio code to drive the debuggers from gui.

[mperillo]

@dimdin

You can run all the tests and filter the coverage that is interesting to you using filters: GitHub - SimonKagstrom/kcov: Code coverage tool for compiled programs, Python and Bash which uses debugging information to collect and report data without special compilation options

Unfortunately it does not work. I’m testing a file in the standard library, so after 19079 you need to test lib/std/std.zig and using test-filter to select the tests to run.

I tried kcov filters, but I was unable to make kcov exclude all the files except the file I want to test.

I tried first to --exclude-path lib followed by --include-path lib/std/<file>.zig, but it results in an empty test coverage.

But thanks for the hint about --clean.

@mperillo

In future I would like to learn how to use a profiler and debugger with Zig.

@dimdin

It’s the same with C.
perf just works for linux and normally you can use your platform specific profiler with zig.
You can use lldb or gdb, or editors such as visual studio code to drive the debuggers from gui.

Zig does have pretty printers for gdb and lldb, but for tracy, the two current bindings are 2 years old. Should I use perf, instead (I’m on Linux)?

Thanks,

[dimdin]

@mperillo

I tried first to --exclude-path lib followed by --include-path lib/std/<file>.zig, but it results in an empty test coverage.

From README:

include everything which contains … but exclude everything that has the … string in it.

--exclude-path lib removes all the included entries.
Use --include-path lib/std/file.zig without --exclude-path.

@mperillo

Should I use perf, instead (I’m on Linux)?

Yes, perf is native linux profiling and tracing.
Awesome free pdf comic for perf: Profiling & tracing with perf
See also: GitHub - brendangregg/FlameGraph: Stack trace visualizer

[dimdin]

@mperillo

By simple I mean that I can use a script to wrap the test command

Another way to run it is:

zig test --test-no-exec -femit-bin=zig-out/bin/tester lib/std/std.zig ...
kcov --clean --include-pattern=lib/std/foo.zig zig-out/cover zig-out/bin/tester

[mperillo]

Thanks @dimdin, the solution was very simple. I was thinking the other way, trying to use --exclude-path.

The missing piece is how to translate a fully qualified name + module (like Build.Step in std) to a filesystem path (e.g lib/std/Build/Step.zig).