Why is this failing with .init_single_threaded?

I refactored this function that was using Child.init() before which was working with .init_single threaded:

pub fn dateChanger(
    self: []const u8,
    io: std.Io,
    allocator: std.mem.Allocator,
    stdout: *std.Io.Writer,
    stderr: *std.Io.Writer,
    file: [][]const u8,
    baseDate: []u8,
    increment_seconds: i64,
    verbose: u8,
) !void {
    for (file) |value| {
        if (std.mem.eql(u8, self, value)) return;
        // Compute new timestamp (convert to seconds, increment, convert back)
        const newDate = try computeIncrementedDate(baseDate, increment_seconds);
        cmd[2] = try std.fmt.allocPrint(allocator, "-DateTimeOriginal={s}", .{newDate});
        defer allocator.free(cmd[2]);
        cmd[3] = try std.fmt.allocPrint(allocator, "-CreateDate={s}", .{newDate});
        defer allocator.free(cmd[3]);
        cmd[4] = try std.fmt.allocPrint(allocator, "-ModifyDate={s}", .{newDate});
        defer allocator.free(cmd[4]);
        cmd[5] = value;

        // exiftool command
        const result = try std.process.run(allocator, io, .{
            .argv = &cmd,
            .stdout_limit = .limited(1024),
            .stderr_limit = .limited(1024),
        });

        defer {
            allocator.free(result.stdout);
            allocator.free(result.stderr);
        }
        switch (verbose) {
            't' => {
                std.log.info("File: {s}", .{value});
                if (result.term.exited == 0) {
                    //STDOUT
                    try stdout.writeAll(result.stdout);
                    try stdout.print("Exit: {d}\n", .{result.term.exited});
                    try stdout.flush();
                } else {
                    //STDERR
                    try stderr.writeAll(result.stderr);
                    try stderr.flush();
                    std.log.err("Exit: {d}", .{result.term.exited});
                }
            },
            'f' => {
                if (result.term.exited == 0)
                    std.log.info("File: {s}", .{value})
                else
                    std.log.err("File: {s}", .{value});
            },
            else => {},
        }
    }
}

I’m trying to understand where does allocation happens that bubbles OOM error but I’m having trouble with it. When I switched to Threaded.init() all worked:

    // Io init
    var io_init = std.Io.Threaded.init(
        allocator,
        .{ .environ = init.environ },
    );
    defer io_init.deinit();
    const io = io_init.io();

See Std.process.spawn new 0.16 help - #7 by Sze

The crux of it is that the .init_single_threaded uses the Failing allocator. All allocations will fail. You are better off using the one from juicy main or creating the full one you need with the designated allocator.

Yes, I understand that. .init_single_threaded does not allow allocations to happen. I’m guessing the spawn process tries to allocate looking at the stack trace, now my question would be the reason behind it:

error: OutOfMemory
/opt/homebrew/Caskroom/zig@nightly/0.17.0-dev.305+bdfbf432d/zig-aarch64-macos-0.17.0-dev.305+bdfbf432d/lib/std/mem
/Allocator.zig:300:82: 0x1002ae077 in allocBytesWithAlignment__anon_9186 (sorter)
    const byte_ptr = self.rawAlloc(byte_count, alignment, return_address) orelse return error.OutOfMemory;
                                                                                 ^
/opt/homebrew/Caskroom/zig@nightly/0.17.0-dev.305+bdfbf432d/zig-aarch64-macos-0.17.0-dev.305+bdfbf432d/lib/std/mem
/Allocator.zig:286:5: 0x100340b1b in allocWithSizeAndAlignment__anon_19289 (sorter)
    return self.allocBytesWithAlignment(alignment, byte_count, return_address);
    ^
/opt/homebrew/Caskroom/zig@nightly/0.17.0-dev.305+bdfbf432d/zig-aarch64-macos-0.17.0-dev.305+bdfbf432d/lib/std/mem
/Allocator.zig:274:55: 0x1003410d3 in allocAdvancedWithRetAddr (sorter)
    const ptr: [*]align(a.toByteUnits()) T = @ptrCast(try self.allocWithSizeAndAlignment(@sizeOf(T), a, n, return_
address));
                                                      ^
/opt/homebrew/Caskroom/zig@nightly/0.17.0-dev.305+bdfbf432d/zig-aarch64-macos-0.17.0-dev.305+bdfbf432d/lib/std/mem
/Allocator.zig:222:21: 0x100341193 in allocWithOptionsRetAddr__anon_19302 (sorter)
        const ptr = try self.allocAdvancedWithRetAddr(Elem, optional_alignment, n + 1, return_address);
                    ^
/opt/homebrew/Caskroom/zig@nightly/0.17.0-dev.305+bdfbf432d/zig-aarch64-macos-0.17.0-dev.305+bdfbf432d/lib/std/mem
/Allocator.zig:252:5: 0x100340f37 in allocSentinel__anon_18425 (sorter)
    return self.allocWithOptionsRetAddr(Elem, n, null, sentinel, @returnAddress());
    ^
/opt/homebrew/Caskroom/zig@nightly/0.17.0-dev.305+bdfbf432d/zig-aarch64-macos-0.17.0-dev.305+bdfbf432d/lib/std/Io/
Threaded.zig:15063:22: 0x100334983 in spawnPosix (sorter)
    const argv_buf = try arena.allocSentinel(?[*:0]const u8, options.argv.len, null);
                     ^
/opt/homebrew/Caskroom/zig@nightly/0.17.0-dev.305+bdfbf432d/zig-aarch64-macos-0.17.0-dev.305+bdfbf432d/lib/std/Io/
Threaded.zig:15235:21: 0x10033321b in processSpawnPosix (sorter)
    const spawned = try spawnPosix(t, options);
                    ^
/opt/homebrew/Caskroom/zig@nightly/0.17.0-dev.305+bdfbf432d/zig-aarch64-macos-0.17.0-dev.305+bdfbf432d/lib/std/pro
cess.zig:444:5: 0x10038eb3f in spawn (sorter)
    return io.vtable.processSpawn(io.userdata, options);
    ^
/opt/homebrew/Caskroom/zig@nightly/0.17.0-dev.305+bdfbf432d/zig-aarch64-macos-0.17.0-dev.305+bdfbf432d/lib/std/pro
cess.zig:498:17: 0x10038a913 in run (sorter)
    var child = try spawn(io, .{
                ^
/Users/j.markovic/GIT/Dario/sorter/src/dateModify.zig:188:24: 0x100386d33 in dateChanger (sorter)
        const result = try std.process.run(allocator, io, .{
                       ^
/Users/j.markovic/GIT/Dario/sorter/src/main.zig:90:5: 0x1003883a7 in main (sorter)
    try dateModify.dateChanger(
    ^
run
└─ run exe sorter failure
error: process exited with error code 1
failed command: /Users/j.markovic/GIT/Dario/sorter/zig-out/bin/sorter -v .

Build Summary: 3/5 steps succeeded (1 failed)
run transitive failure
└─ run exe sorter failure

My naive understanding is that it allocates the args as null-terminated form which the kernel expects. There may be other allocations, but that is the culprit here. You can see that in the stacktrace:

    const argv_buf = try arena.allocSentinel(?[*:0]const u8, options.argv.len, null);

It’s allocating the argv buf with sentinel terminated values.

Even though I’m passing a pointer to them. Interesting. I need to learn how to follow vtables while reading the source code.

Yes, even if you are passing a pointer to them. Your pointer is probably a regular Zig slice (with out the sentinel). Spawn will create a new slice for them that has the sentinel and then copy them over.

1 Like

Yup, just took me a while to find it in Threaded.spawnPosix():

var arena_allocator = std.heap.ArenaAllocator.init(t.allocator);
defer arena_allocator.deinit();
const arena = arena_allocator.allocator();

// The POSIX standard does not allow malloc() between fork() and execve(),
// and this allocator may be a libc allocator.
// I have personally observed the child process deadlocking when it tries
// to call malloc() due to a heap allocation between fork() and execve(),
// in musl v1.1.24.
// Additionally, we want to reduce the number of possible ways things
// can fail between fork() and execve().
// Therefore, we do all the allocation for the execve() before the fork().
// This means we must do the null-termination of argv and env vars here.
const argv_buf = try arena.allocSentinel(?[*:0]const u8, options.argv.len, null);
1 Like