Hello,
I have been playing around with implementing cat-like utility (mostly to help other people learn).
While doing so i encountered interesting performance characteristic of sendFile() on linux.
I will focus on the static builds, libC builds just link libC and use its allocator.
I am cating a 0.5 GiB file with stdout redirected to a file.
I checked that the code does as expected and counted the syscalls used.
The
copy_file_range is a bit suspicious, but I have no idea where it came from.
Code:
const std = @import("std");
// this is not 1:1 mapping of the C code, you could do that but its more common to go the cross platfrorm way
// this implementation will work on single threaded blocking code or with green threads executor (just change the io)
pub fn main(init: std.process.Init.Minimal) !void {
// create a threaded io
var single_threaded = std.Io.Threaded.init_single_threaded;
const io = single_threaded.io();
// this is really bad general purpose allocator, but its the best we have for single-threaded builds
const gpa = std.heap.page_allocator;
// allocate the args as we itterate, on linux it does nothing but for example windows needs them allocated
var it = try init.args.iterateAllocator(gpa);
// dont forget to free them
defer it.deinit();
// skip the executable name
_ = it.skip();
// get the first next arguemnt
var arg: ?[:0]const u8 = it.next();
// check if we got any (without this check we could have simpler while below, but no real perf lost here)
if (arg == null) return error.NoArgumentGiven;
// setup the buffer for writing (read per 16384 byte chunks), only relevant on sendFile fallback
var stdout_buf: [16384]u8 = undefined;
// setup the buffer for reading (read per 16384 byte chunks)
var file_buf: [16384]u8 = undefined;
// cwd is static, does not need to be closed, check its doc comment
const local_dir: std.Io.Dir = .cwd();
// while we get an argument, unwrap it from optional to name, at the end of loop get next
while (arg) |name| : (arg = it.next()) {
// open the file in cwd as read only
const file = try local_dir.openFile(io, name, .{ .mode = .read_only });
// dont forget to close it
defer file.close(io);
// create the reader backed by the buffer
var file_reader = file.reader(io, &file_buf);
// pull out the reader interface
const reader = &file_reader.interface;
// create the writer backed by the buffer (you dont close stdout, same as C)
var stdout_writer = std.Io.File.stdout().writer(io, &stdout_buf);
// pull out the writer interface
const writer = &stdout_writer.interface;
// just let it stream the data from one another till it hits EOF, discard the ammount transfered
_ = try reader.streamRemaining(writer);
// dont forget to flush
try writer.flush();
}
}
const std = @import("std");
// this is not 1:1 mapping of the C code, you could do that but its more common to go the cross platfrorm way
// this implementation will work on single threaded blocking code or with green threads executor (just change the io)
pub fn main(init: std.process.Init.Minimal) !void {
// create a threaded io
var single_threaded = std.Io.Threaded.init_single_threaded;
const io = single_threaded.io();
// this is really bad general purpose allocator, but its the best we have for single-threaded builds
const gpa = std.heap.page_allocator;
// allocate the args as we itterate, on linux it does nothing but for example windows needs them allocated
var it = try init.args.iterateAllocator(gpa);
// dont forget to free them
defer it.deinit();
// skip the executable name
_ = it.skip();
// get the first next arguemnt
var arg: ?[:0]const u8 = it.next();
// check if we got any (without this check we could have simpler while below, but no real perf lost here)
if (arg == null) return error.NoArgumentGiven;
// setup the buffer for writing (read per 16384 byte chunks), only relevant on sendFile fallback
var stdout_buf: [16384]u8 = undefined;
// while we get an argument, unwrap it from optional to name, at the end of loop get next
while (arg) |name| : (arg = it.next()) {
// cwd is static, does not need to be closed, check its doc comment
const local_dir: std.Io.Dir = .cwd();
// open the file in cwd as read only
const file = try local_dir.openFile(io, name, .{ .mode = .read_only });
// dont forget to close it
defer file.close(io);
// create the writer backed by the buffer (you dont close stdout, same as C) stdout is streaming FD, so you save 1 syscall before it figures it out
var stdout_writer = std.Io.File.stdout().writerStreaming(io, &stdout_buf);
// pull out the writer interface
const writer = &stdout_writer.interface;
// create the reader backed by no buffer, the sendfile fills the writer buffer directly
var file_reader = file.reader(io, &.{});
// no need to pull out the reader interface, sendFile takes the File.Reader not Io.Reader
// this attempts to use more optimal syscalls, if not aviable it falls back to streaming (single buffered one tho.)
// just let it stream the data from one another till it hits EOF, discard the ammount transfered
_ = try writer.sendFileAll(&file_reader, .unlimited);
// dont forget to flush
try writer.flush();
}
}
Its compiled with ReleaseFast, native architecture and:
.single_threaded = true,
.unwind_tables = .none,
.strip = true,
Binaries are basically the same size aswell, the sendFIle being larger by like 580 bytes.
My expectation would be that the sendFile would be faster, anyone have any ideas why is it like this?
Robert 