Is the standard tar archive extractor good enough?

Help
,
1 
fn extract_tarxz_to_dir(allocator: std.mem.Allocator, outDir: std.fs.Dir, file: std.fs.File) !void {
    var decompressed = try std.compress.xz.decompress(allocator, file.reader());
    defer decompressed.deinit();
    try std.tar.pipeToFileSystem(outDir, decompressed.reader(), .{ .mode_mode = .ignore });
}

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer std.debug.assert(gpa.deinit() == .ok);
    const allocator = gpa.allocator();

    const testTar = "/home/user/softwares/zig-linux-x86_64-0.12.0-dev.2271+14efbbfd8.tar.xz";
    const testTarFile = try std.fs.openFileAbsolute(testTar, .{});
    const testDir = try std.fs.openDirAbsolute("/home/user/softwares/zig-linux-x86_64-0.12.0-dev.2271+14efbbfd8", .{});
    try extract_tarxz_to_dir(allocator, testDir, testTarFile);
}

https://ziglang.org/documentation/master/std/#A;std:tar.pipeToFileSystem
i am not sure the performance seems to be so bad for some reason, am i doing something wrong? does the zig compiler use the same implementation to extract tar archives of dependency modules while building?

2

Welcome to the forum!

Yes, the Zig compiler uses std.tar as part of its own package fetching code: zig/src/Package/Fetch.zig at 7d81c952d57a76454c31b13b3ec8e21388f02171 · ziglang/zig · GitHub It also uses std.compress.xz to decompress xz-compressed archives: zig/src/Package/Fetch.zig at 7d81c952d57a76454c31b13b3ec8e21388f02171 · ziglang/zig · GitHub

How are you measuring the performance of this example? If you’re building an executable to run using zig build-exe, you can add -OReleaseFast to build with safety checks disabled and optimizations for speed; the default if you don’t specify anything is -ODebug, which is good for debugging due to having safety checks enabled and faster compilation, but not good for benchmarking.

Edit: with that said, I tried out the example you shared, and it does seem to be considerably slower than running tar -xf directly on the archive.

Edit 2: aside from the suggestion above on the build mode (and the fact that the current std.compress.xz implementation may not be as optimized as it could be), a couple more suggestions for your test code:

  1. std.heap.GeneralPurposeAllocator tends to be rather slow. For best performance, the best choice right now might be std.heap.c_allocator, which also requires linking libc via the -lc option to zig build-exe.
  2. It’s almost always a good idea to use a buffered reader (std.io.BufferedReader) when reading from a file, since this can greatly reduce the number of expensive syscalls needed to fully read and process the file.

With these suggestions, here’s what the example might look like:

const std = @import("std");

fn extract_tarxz_to_dir(allocator: std.mem.Allocator, outDir: std.fs.Dir, file: std.fs.File) !void {
    var buffered_reader = std.io.bufferedReader(file.reader());
    var decompressed = try std.compress.xz.decompress(allocator, buffered_reader.reader());
    defer decompressed.deinit();
    try std.tar.pipeToFileSystem(outDir, decompressed.reader(), .{ .mode_mode = .ignore });
}

pub fn main() !void {
    const allocator = std.heap.c_allocator;

    const testTar = "/var/home/ian/tmp/zig-linux-x86_64-0.12.0-dev.2271+14efbbfd8.tar.xz";
    const testTarFile = try std.fs.openFileAbsolute(testTar, .{});
    const testDir = try std.fs.openDirAbsolute("/var/home/ian/tmp/zig-linux-x86_64-0.12.0-dev.2271+14efbbfd8", .{});
    try extract_tarxz_to_dir(allocator, testDir, testTarFile);
}

This is still considerably slower than tar -xf, though, so I’m sure there are opportunities for std.compress.xz to be further optimized.

4 Likes
3

Hi there, thanks for the wholesome welcome and time to look out for this naive question.

So, the bottleneck here seems to be the xz decompress implementation I see.
Ow and was wondering if the std.heap.c_allocator is cross-platform kinda confused by Have to link with libc comment.

4

c_allocator is libc’s malloc so, yes, it’s cross-platform as all the 3 big OSs have at least one libc implementation.

But using it requires depending on libc, something that Zig doesn’t do by default. In practice it means adding exe.linkLibC() to your build.zig (or adding -lc to the cli if you’re not using a build script)

1 Like
5

I thought of this today when I saw this PR, which was just merged: compress.xz: fix slow running read loop by ianic · Pull Request #19289 · ziglang/zig · GitHub

With this merged (Zig version 0.12.0-dev.3298+32f602ad1), updating my adapted example above to use zig-linux-x86_64-0.12.0-dev.3284+153ba46a5.tar.xz (the current master download available on ziglang.org), the gap has narrowed considerably:

$ time tar -xf zig-linux-x86_64-0.12.0-dev.3284+153ba46a5.tar.xz 

real    0m3.210s
user    0m2.647s
sys     0m1.730s

$ time ./test

real    0m6.210s
user    0m4.551s
sys     0m1.600s

Awesome job to ianic (doesn’t look like he’s on this forum, unfortunately) for this improvement :tada:

6 Likes