Does Zig's directory walking have a hard cap? Edit: No, either ghosts or NFS oddities

I’m trying to walk a directory iteratively in Zig using the standard library. I open a directory using std.fs.cwd().openDir(path, .{.iterate = true}), and then make a call to ‘walk’.

fn iter_table_dir(allocator: Allocator, dir: std.fs.Dir) !void {
    var walk = try dir.walk(allocator);
    defer walk.deinit();
    while (try walk.next()) |entry| {
        if (entry.kind == .file) {
            const dirname = entry.path[0 .. entry.path.len - entry.basename.len];
            if (std.mem.endsWith(u8, entry.basename, "TableB_en.txt")) {
                try stdout.print("->{s} {s}\n", .{ dirname, entry.basename });
            } else {
                try stdout.print("{s} {s}\n", .{ dirname, entry.basename });
            }
        } else {
            try stdout.print("{s}\n", .{entry.path});
        }
    }
}

pub fn main() !void {
    var arena_state = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena_state.deinit();
    const arena = arena_state.allocator();

    const args = try std.process.argsAlloc(arena);

    if (args.len != 3) fatal("wrong number of arguments\n", .{});

    var input_table_dir = try std.fs.cwd().openDir(
        args[1],
        .{ .iterate = true },
    );
    defer input_table_dir.close();

    var output_code_dir = try std.fs.cwd().openDir(args[2], .{});
    defer output_code_dir.close();

    try iter_table_dir(arena, input_table_dir);

    return std.process.cleanExit();
}

I am encountering a strange issue, however, where after iterating through 17-18 directories, all other directories are reported as empty. I have 28 directories, with 5 files in each directory. If I pass the path to one of the “empty” directories to my executable, it then shows file inside.

Am I doing something wrong, or is there a bug in the directory walking library code?
Some output from my code…

44
44/ BUFRCREX_CodeFlag_en.txt
->44/ BUFRCREX_TableB_en.txt
44/ BUFR_TableA_en.txt
44/ BUFR_TableC_en.txt
44/ BUFR_TableD_en.txt
34
34/ BUFRCREX_34_0_0_CodeFlag_en.txt
->34/ BUFRCREX_34_0_0_TableB_en.txt
34/ BUFR_34_0_0_TableA_en.txt
34/ BUFR_34_0_0_TableC_en.txt
34/ BUFR_34_0_0_TableD_en.txt
18
18/ BUFRCREX_18_1_0_CodeFlag_en.txt
->18/ BUFRCREX_18_1_0_TableB_en.txt
18/ BUFR_18_1_0_TableA_en.txt
18/ BUFR_18_1_0_TableC_en.txt
18/ BUFR_18_1_0_TableD_en.txt
19
19/ BUFRCREX_19_1_1_CodeFlag_en.txt
->19/ BUFRCREX_19_1_1_TableB_en.txt
19/ BUFR_19_1_1_TableA_en.txt
19/ BUFR_19_1_1_TableC_en.txt
19/ BUFR_19_1_1_TableD_en.txt
20
20/ BUFRCREX_20_0_0_CodeFlag_en.txt
->20/ BUFRCREX_20_0_0_TableB_en.txt
20/ BUFR_20_0_0_TableA_en.txt
20/ BUFR_20_0_0_TableC_en.txt
20/ BUFR_20_0_0_TableD_en.txt
21
21/ BUFRCREX_21_0_0_CodeFlag_en.txt
->21/ BUFRCREX_21_0_0_TableB_en.txt
21/ BUFR_21_0_0_TableA_en.txt
21/ BUFR_21_0_0_TableC_en.txt
21/ BUFR_21_0_0_TableD_en.txt
22
22/ BUFRCREX_22_0_1_CodeFlag_en.txt
->22/ BUFRCREX_22_0_1_TableB_en.txt
22/ BUFR_22_0_1_TableA_en.txt
22/ BUFR_22_0_1_TableC_en.txt
22/ BUFR_22_0_1_TableD_en.txt
23
23/ BUFRCREX_23_0_0_CodeFlag_en.txt
->23/ BUFRCREX_23_0_0_TableB_en.txt
23/ BUFR_23_0_0_TableA_en.txt
23/ BUFR_23_0_0_TableC_en.txt
23/ BUFR_23_0_0_TableD_en.txt
24
24/ BUFRCREX_24_0_0_CodeFlag_en.txt
->24/ BUFRCREX_24_0_0_TableB_en.txt
24/ BUFR_24_0_0_TableA_en.txt
24/ BUFR_24_0_0_TableC_en.txt
24/ BUFR_24_0_0_TableD_en.txt
25
25/ BUFRCREX_25_0_0_CodeFlag_en.txt
->25/ BUFRCREX_25_0_0_TableB_en.txt
25/ BUFR_25_0_0_TableA_en.txt
25/ BUFR_25_0_0_TableC_en.txt
25/ BUFR_25_0_0_TableD_en.txt
26
26/ BUFRCREX_26_1_0_CodeFlag_en.txt
->26/ BUFRCREX_26_1_0_TableB_en.txt
26/ BUFR_26_1_0_TableA_en.txt
26/ BUFR_26_1_0_TableC_en.txt
26/ BUFR_26_1_0_TableD_en.txt
27
27/ BUFRCREX_27_0_0_CodeFlag_en.txt
->27/ BUFRCREX_27_0_0_TableB_en.txt
27/ BUFR_27_0_0_TableA_en.txt
27/ BUFR_27_0_0_TableC_en.txt
27/ BUFR_27_0_0_TableD_en.txt
28
28/ BUFRCREX_28_0_0_CodeFlag_en.txt
->28/ BUFRCREX_28_0_0_TableB_en.txt
28/ BUFR_28_0_0_TableA_en.txt
28/ BUFR_28_0_0_TableC_en.txt
28/ BUFR_28_0_0_TableD_en.txt
29
29/ BUFRCREX_29_0_0_CodeFlag_en.txt
->29/ BUFRCREX_29_0_0_TableB_en.txt
29/ BUFR_29_0_0_TableA_en.txt
29/ BUFR_29_0_0_TableC_en.txt
29/ BUFR_29_0_0_TableD_en.txt
30
30/ BUFRCREX_30_0_0_CodeFlag_en.txt
->30/ BUFRCREX_30_0_0_TableB_en.txt
30/ BUFR_30_0_0_TableA_en.txt
30/ BUFR_30_0_0_TableC_en.txt
30/ BUFR_30_0_0_TableD_en.txt
31
31/ BUFRCREX_31_0_0_CodeFlag_en.txt
->31/ BUFRCREX_31_0_0_TableB_en.txt
31/ BUFR_31_0_0_TableA_en.txt
31/ BUFR_31_0_0_TableC_en.txt
31/ BUFR_31_0_0_TableD_en.txt
32
32/ BUFRCREX_32_0_0_CodeFlag_en.txt
->32/ BUFRCREX_32_0_0_TableB_en.txt
32/ BUFR_32_0_0_TableA_en.txt
32/ BUFR_32_0_0_TableC_en.txt
32/ BUFR_32_0_0_TableD_en.txt
33
33/ BUFRCREX_33_0_0_CodeFlag_en.txt
->33/ BUFRCREX_33_0_0_TableB_en.txt
33/ BUFR_33_0_0_TableA_en.txt
33/ BUFR_33_0_0_TableC_en.txt
33/ BUFR_33_0_0_TableD_en.txt
35
36
37
38
39
40
41
42
43

It seems like everything is okey with code and it works without issues on my machine.
Maybe you have strange symbols in file names or some files are link?
Can you give more information about your setup.

Can copy file subtree and try to mimimize it to get nice short reproduction?

Well, I came back to give some more details on my system/file structure, only for the problem to have resolved itself within the last half hour. This is truly bizarre.

The only thing I can guess is that it is due to the filesystem being an NFS mount? Perhaps some sort of syncing issue there? I don’t control the system I’m building on, but the home directory of my RHEL8 machine is mounted via NFS. What is strange is typical OS command line tools showed the files via “ls”… so the files were absolutely present. Is there something behind the scenes that would cause interaction with an NFS mount to behave this way, or am I just doomed to blaming it on ghosts? :sweat_smile:

1 Like

I cannot provide any specific details now, but I clearly remember seeing in the past directory walking issues in NFS mount points, so not surprised at all about the symptoms you are seeing.