While loop `continue` bug?

Absolutely lost on this one:

I switched to std.fs Walker but continue does not work as expected:

while (try walker.next()) |entry| {
    if (Util.isIgnored(entry.basename, ignore_items)) continue;

    switch (entry.kind) {
        .file => {
            const src_path = try std.fs.path.join(
                allocator,
                &.{ source_with_slash, entry.path },
            );

            const dest_path = try std.fs.path.join(
                allocator,
                &.{ dest_with_slash, entry.path },
            );

            const file = Dotfile.new(src_path, dest_path);

            try files.append(allocator, file);
        },
        else => continue,
    }
}

Switch is being evaluated despite true from isIgnored().

I know isIgnored() is not the issue since replacing continue with break works as expected.

zig 0.14.0 (FBSD)

I tried to switch everything to null-terminated strings, but isIgnored() works just fine as it is.

Confirmation

if (Util.isIgnored(entry.basename, ignore_items)) {
   std.debug.print("IGNORED: {}\n", .{std.zig.fmtEscapes(entry.basename)});
   continue;
}
IGNORED: README.md
IGNORED: .gitmodules
IGNORED: .git
IGNORED: .github
IGNORED: .gitignore
{"total":51,"updated":43,"template":0,"render":47,"binary":4,"errors":0,"dry_run":false}

but switch is still being evaluated despite continue

total 447
drwxr-xr-x   6 charlie charlie   14B Sep 19 14:32 .
drwxr-xr-x  41 charlie charlie   81B Sep 19 14:33 ..
-rw-r--r--   1 charlie charlie   27B Sep 19 14:32 .bashrc
-rw-r--r--   1 charlie charlie  377K Sep 19 14:32 .face.icon
drwxr-xr-x   7 charlie charlie   12B Sep 19 14:32 .git
drwxr-xr-x   3 charlie charlie    4B Sep 19 14:32 .github
drwxr-xr-x   2 charlie charlie    3B Sep 19 14:32 .gnupg

walk recursively iterates through all elements of a folders, that includes files in subdirectories.
You are ignoring the folder named .git but you are still iterating through the contents of these folders.
Your Util.isIgnored will return false for e.g. .git/config, and so the switch gets executed for it, which I assume leads to the recreation of the .git folder later in the code.

not sure this is the case, im getting the same result if i refactor to

while (try walker.next()) |entry| {
    switch (entry.kind) {
        .file => {
            const src_path = try std.fs.path.join(
                allocator,
                &.{ source_with_slash, entry.path },
            );

            const dest_path = try std.fs.path.join(
                allocator,
                &.{ dest_with_slash, entry.path },
            );

            const file = Dotfile.new(src_path, dest_path);

            if (!Util.isIgnored(entry.basename, ignore_items)) try files.append(allocator, file);
        },
        else => continue,
    }
}

Looks like it can be brute-forced:

walk: while (try walker.next()) |entry| {
    if (Util.isIgnored(entry.basename, ignore_items)) {
        if (entry.kind = .directory) {
            // remove from stack, with prejudice
            var item = walker.stack.pop().?;
            // note: don't let this be the root directory
            item.iter.dir.close();
        }
        continue :walk;
    }

    switch (entry.kind) {
        .file => {
            const src_path = try std.fs.path.join(
                allocator,
                &.{ source_with_slash, entry.path },
            );

            const dest_path = try std.fs.path.join(
                allocator,
                &.{ dest_with_slash, entry.path },
            );

            const file = Dotfile.new(src_path, dest_path);

            try files.append(allocator, file);
        },
        else => continue :walk,
    }
}

I haven’t tried this code but that’s the basic idea.

It’s pretty ugly stuff though, it would be better to encapsulate in a function like walker.skipThisDirectory(). Seems like skipping directories during a recursive iteration is the kind of thing which will happen.

1 Like

oh thanks! this works as expected:

total 448
drwxr-xr-x   7 charlie charlie   15B Sep 19 15:21 .
drwxr-xr-x  41 charlie charlie   81B Sep 19 15:21 ..
-rw-r--r--   1 charlie charlie   27B Sep 19 15:21 .bashrc
drwxr-xr-x  33 charlie charlie   36B Sep 19 15:21 .config
-rw-r--r--   1 charlie charlie  377K Sep 19 15:21 .face.icon
drwxr-xr-x   2 charlie charlie    3B Sep 19 15:21 .gnupg
-rw-r--r--   1 charlie charlie  435B Sep 19 15:21 .gtkrc-2.0.mine
-rw-r--r--   1 charlie charlie  402B Sep 19 15:21 .profile
-rw-r--r--   1 charlie charlie  118B Sep 19 15:21 .shrc
-rw-r--r--   1 charlie charlie   57B Sep 19 15:21 .xonshrc
-rw-r--r--   1 charlie charlie  101B Sep 19 15:21 .zprofile
drwxr-xr-x   4 charlie charlie    4B Sep 19 15:21 .zsh-custom
-rw-r--r--   1 charlie charlie  268B Sep 19 15:21 .zshrc
drwxr-xr-x   2 charlie charlie   13B Sep 19 15:21 bin
drwxr-xr-x   3 charlie charlie    4B Sep 19 15:21 misc

maybe I should open a PR for the Walker, feels like this is a very elaborate approach for a simple skip task.

1 Like

Here is a related skip function for the dir walker:

3 Likes

swell, I’ll check it out right after fixing realpathAlloc(). turned out its a big nono now, but the bug notice is only for realpath().

See my comment in that thread and the linked PR.

AFAICT your only use of realpath is to convert paths to absolute, but there’s no need for that. The std.fs.Dir functions all work with any path type (you can use std.fs.cwd() to get a Dir to start with if you need it).

The cwd() approach will not work if the user is calling the application from the parent dir unrelated to relative paths in the app config. And my goal is to support both relative and absolute paths in the config (in some cases an absolute path in the config is dangerous). It works for me right now because I only handle relative paths in test configs. But things will change soon.

1 Like

Two things:

  1. realpath already has an implicit dependency on the CWD. When you call realpath("foo") it will resolve it relative to the CWD.
const std = @import("std");

pub fn main() !void {
    var debug_allocator = std.heap.DebugAllocator(.{}){};
    defer std.debug.assert(debug_allocator.deinit() == .ok);
    const allocator = debug_allocator.allocator();

    const some_path = "foo";
    const abs_path = try std.fs.realpathAlloc(allocator, some_path);
    defer allocator.free(abs_path);

    std.debug.print("{s}\n", .{abs_path});
}
~/Programming/zig/tmp$ ./foorealpath
/home/ryan/Programming/zig/tmp/foo

~/Programming/zig/tmp$ cd subdir

~/Programming/zig/tmp/subdir$ ../foorealpath
/home/ryan/Programming/zig/tmp/subdir/foo
  1. It sounds like you might want to use std.fs.selfExeDirPath as the starting point when resolving the paths:
const std = @import("std");

pub fn main() !void {
    var debug_allocator = std.heap.DebugAllocator(.{}){};
    defer std.debug.assert(debug_allocator.deinit() == .ok);
    const allocator = debug_allocator.allocator();

    const self_dir_path = try std.fs.selfExeDirPathAlloc(allocator);
    defer allocator.free(self_dir_path);

    std.debug.print("self dir path: {s}\n", .{self_dir_path});

    var root_dir = try std.fs.cwd().openDir(self_dir_path, .{});
    defer root_dir.close();

    // Always will open the `foo` relative to the exe location
    var some_file = try root_dir.openFile("foo", .{});
    defer some_file.close();
}
~/Programming/zig/tmp$ ./fooselfexedir
self dir path: /home/ryan/Programming/zig/tmp

~/Programming/zig/tmp$ cd subdir

~/Programming/zig/tmp/subdir$ ../fooselfexedir
self dir path: /home/ryan/Programming/zig/tmp
1 Like

oh wow, good to know, thanks!
tho not sure exe path can help on this one, common install locations like opt and etc. are just as unrelated to homedir as cwd might be.

Then selfExeDirPath might not be the right solution, but the general idea applies:

  • Get the path to the root directory you want to resolve the configurable/user-inputted paths from
  • Open that directory
  • Call std.fs.Dir functions using that directory

This library might be of interest:

1 Like

I saw that one while thinking about ~ expansion. maybe ill just support paths with env vars instead (like $HOME/foo). That way I will have a concrete root directory as a starting point.

something like

pub fn pathFormat(
    allocator: std.mem.Allocator,
    path: []const u8,
) ![]const u8 {
    const trailing_slash = std.mem.endsWith(u8, path, "/");

    if (std.mem.startsWith(u8, path, "$HOME")) {
        const home = try getXdgDir(allocator, XdgDir.Home);
        defer allocator.free(home);

        const size = std.mem.replacementSize(u8, path, "$HOME", home);
        const new_path = try allocator.alloc(u8, size);
        defer allocator.free(new_path);

        _ = std.mem.replace(u8, path, "$HOME", home, new_path);

        const target = if (trailing_slash)
            new_path
        else
            try std.fmt.allocPrint(allocator, "{s}/", .{new_path});

        defer if (trailing_slash) allocator.free(target);

        return target;
    } else if (!trailing_slash) {
        return try std.fmt.allocPrint(allocator, "{s}/", .{path});
    } else {
        return path;
    }
}

Pretty rough, but it will do for now

Since you mentioned my custom walker, I just want to give an update: I ended up rewriting it in a way which is the opposite of the default (and thus I have the opposite to skip). Instead of skipping directories when I notice I didn’t need to open them, I added an accept() function to traverse into the directory only when I need to. Might be a silly micro-optimization, but reduced openat and close syscalls by about 10%.

This is also a bit more in-line with the unmanaged data structures since I’m always calling it in the same location where I have the allocator, so I don’t need to store it in the walker struct.

I hope this piece of update helps in any way :folded_hands:

6 Likes

That’s a pretty nice design, I think I’d be in favor of that being upstreamed.

(I’d probably bikeshed the names of accept/skip, though, maybe something like recurseIntoDir/skipContainingDir :upside_down_face:)

If you’re interested in making a PR, my suggestion would be to:

3 Likes

That’s flattering and honestly a good idea. I’ll look into it!

1 Like