Integration tests are not covered on CI runners

[charlie]

I implemented kcov for my test suite, but integration tests are ignored on GitHub runners. It works locally tho xtxf/build.zig at trunk · charlesrocket/xtxf · GitHub

Tried with xvfb, but no positive results. Not sure if this is termbox or I am doing something else wrong.

[charlie]

Seems like I forgot to handle tb failures properly

[charlie]

Added debug mode to avoid termbox calls—still no CI output test: check `stderr` · charlesrocket/xtxf@1e02585 · GitHub

its getting weird

[dimdin]

Looks like it is running a different version of default cli test.

Try mlugg/setup-zig@v1 in ci.yml instead of goto-bus-stop/setup-zig@v2

Github action thread in ziggit:

GitHub Actions: mlugg/setup-zig

Hey everyone! I’ve spent a little time putting together a GitHub Action to use for creating Actions workflows utilizing the Zig compiler. You can find it here. The typical use case here is setting up CI for Zig projects. Today, people often use a similar Action by goto-bus-stop. My new Action is an alternative which makes (in my opinion) a few improvements. Zig is not fetched from ziglang.org, but rather from a third-party mirror. This decision was made to avoid frequent downloads from the of…

---

EDIT:

I just found what it runs:

charlesrocket/xtxf/blob/1e025859a3a35895d07a831626831fd09e68e85a/test/cli.zig#L36-L46

First of all expect testing functions first argument must be the expected value and the second argument must be the actual value.

Change this:

    try std.testing.expectStringEndsWith(proc.err, default_run);
    try std.testing.expectEqual(proc.term.Exited, 0);

to:

    try std.testing.expectStringEndsWith(default_run, proc.err);
    try std.testing.expectEqual(0, proc.term.Exited);

---

I cannot find default_run declaration!

[charlie]

Thanks! Thats interesting, i’ll try replacing the action.

@dimdin

First of all expect testing functions first argument must be the expected value and the second argument must be the actual value.

O yes, that would be somewhat critical here , missed it.

L215 is default_run

[dimdin]

OK, now it makes sense.

The facts are:

• The executable runs.
• The error level of the executable is 0.
• There is no output in stderr.
• log.info statements are .xtxf scoped and the default log level includes info.
• No log.info statements executed.

I am guessing that log.info statements are not executed because the CI machine have no actual terminal.
Simply adding an unconditional log.info statement after main is enough to validate this.

[charlie]

What about the coverage run tho? It gets the output but coverage is ignored test: check `stderr` · charlesrocket/xtxf@1e02585 · GitHub

I don’t wan to pollute outputs on live runs so trying to use debug mode for logs

[dimdin]

The coverage run zig build coverage looks good.
Adding --verbose after zig build is a good idea to have the commands that run.

---

The coverage merge is "kcov", "--merge", "kcov-out", "kcov-test", "kcov-unit"
That means store the output in kcov-out.
Note that codecov github action directory is configured as kcov-out/kcov-merged.

[charlie]

Thats the merge dir, ill check the parent dir but current value seems to be valid (I can see valid coverage locally). Replaced the action, but no changes.

@dimdin

Change this:

    try std.testing.expectStringEndsWith(proc.err, default_run);
    try std.testing.expectEqual(proc.term.Exited, 0);

to:

    try std.testing.expectStringEndsWith(default_run, proc.err);
    try std.testing.expectEqual(0, proc.term.Exited);

Looks like its correct

UPD
So right now tests are with no output and failing with debug mode while coverage run is ignoring debug flag in tests (???), has output but coverage is ignored, probably due termbox call. Makes zero sense to me, coverage is using same integration tests.

[charlie]

Looks like no output is due non-debug builds (only debug passes, small/etc has no output). I just need to make sure logging is happening in all releases, not just debug. Coverage is missing probably due non-debug build as well, at least this makes some sense.

[dimdin]

The default log level depends on built mode.
You can use the std_options–as pub–in your root module to set the log level:

pub const std_options: std.Options = .{
    .log_level = .info,
};

[charlie]

Yep, just added it and integration tests are now with output, tho looks like I need to adjust debug subroutine in the handler (sleep value)—some output is too short fix: set std options · charlesrocket/xtxf@1f2e857 · GitHub

[charlie]

CI is passing now, but coverage is still invalid, switching upload dir in coverage action not helping. Exceptionally weird case.

test: add `debug` mode