Getting the length of the stdin buffer on Linux

Hello everyone,

i am new to zig and as a learning project i decided to work on my own posix utilities.

Currently working on cat and i am using the following code to write the stdin content to stdout:

    const stdin = std.Io.File.stdin();
    const stdin_size = try stdin.length(io);
    const buf: []u8 = try allocator.alloc(u8, stdin_size);
    defer allocator.free(buf);
    var reader = stdin.reader(io, buf);
    _ = try reader.interface.readSliceAll(buf);
    
    try std.Io.File.stdout().writeStreamingAll(io, buf);

On my OpenBSD machine this works like a charm, but on Linux stdin.length always returns 0, even when i am piping data into my program.

Is there a cross platform way of getting the length of stdin, so that i know how much memory i have to allocate?

(i am using zig 0.17.0)

I’ll hazard a guess that the answer is mostly no, or at least not consistently. e.g. on Linux unless you know stdin happens to be a file it won’t work. For a pipe, terminal, etc. you’ll get the 0 you’re seeing here until you read the whole stream.

Can you make that assumption?

both Linux and BSD support direct file-to-file copies, so you can avoid reading the data into memory and therefore avoid needing to know how much memory to allocate

The reader.stream* functions will try to use that syscall(s) (rather specific implementations will), and if you give the writer (stream functions take a writer) a sizeable buffer it will use that on platforms that don’t have such syscall(s).

Though it may not be the most optimal it should be decent, you can always use lower level API’s if you need.

Blockquote
The reader.stream* functions

Thanks for the Information, i did not know that exists.

I tried to test this by using the stream function of the reader i got from Std.Io.File.stdin().reader, but the stream function is private on Io.File.Reader, so i tried to use streamMode instead, but it expects a Io.Writer and my stdout writer is a Io.File.Writer so i guess that doesnt work either?

I don’t even know what the difference between Io.Writer and Io.File.Writer is. I guess i should stop wasting everyone’s time here and start studying the standard library source code, because it is just a black box for me.

But what i was wondering: wouldn’t the stream function just return a EndOfStream Error if the buffer i assigned to the reader is bigger than the reader stream? Which means i would have to allocate memory anyways, because i do not know the size at compiletime.

Io.Writer is an interface that wraps whatever writer implementation you need to use, and allows code to be more easily reusable.
Io.File.Reader.stream is private because you aren’t supposed to use it directly. Io.File.Writer/Reader is a specific implementation of the Io.Writer/Reader interface that operates on files (in this case, Io.File.stdout() and Io.File.stdin()). To access the interface from the Io.File.Writer, as you’ve done in the first example, you can use const stdout_w: *Io.Writer = &stdout_writer.interface.

Here’s an example of (as far as I know) idiomatic code that would do what you want (cat):

// stdin
var stdin_buffer: [512]u8 = undefined; // buffer size is arbitrary but this is what I usually use
var stdin_reader: Io.File.Reader = Io.File.stdin().reader(io, &stdin_buffer); // create stdin reader
const stdin_r: *Io.Reader = &stdin_reader.interface; // access stdin_reader's interface by pointer
// stdout
var stdout_buffer: [512]u8 = undefined;
var stdout_writer: Io.File.Writer = Io.File.stdout().writer(io, &stdout_buffer); // create stdout writer
const stdout_w: *Io.Writer = &stdout_reader.interface; // access stdout_writer's interface by pointer

// stream stdin to stdout
_ = try stdin_r.streamRemaining(stdout_w);
try stdout_w.flush(); // I don't know if this is necessary

In the case of the reader.stream* functions, they aren’t implemented as reading all of stdin into the buffer and then writing it all together to stdout. On Linux, Io.File.Reader can implement them as a direct file-to-file copy as was previously mentioned, and when that isn’t available, it can still do a naïve “read-chunk, write-chunk” loop (in units of the buffer size) until the file is fully copied. None of this requires allocating space for the entire file, and only the space in your provided buffer is ever needed by the program.

Conceivably, you could implement cat by allocating space for all of stdin and writing it to stdout, which, although not very efficient, might use Io.Writer.Allocating. This happens to lead to using the _ = try stdin.streamRemaining(&alloc_writer.writer) code again (.writer accesses the allocating writer’s interface) to allocate stdin’s contents.

On Linux, stdin also returns the size. I would have been surprised if it didn’t; the function simply passes the value through directly from the OS. So the question is: how did you call the function on Linux? For example, if you execute zig run cat.zig < cat.zig, everything works as expected.

Io.Writer is the type of the field interface in Io.File.Writer. You access the functions that implement the interface through this field:

var reader = std.Io.File.stdin().reader(io, reader_buf);
var writer = std.Io.File.stdout().writer(io, writer_buf);
const size = try reader.interface.streamRemaining(&writer.interface);

I don’t think “wraps” is a good metaphor here. We are not in OOP lands with inheritance hierarchies that hide the implementation in a private field of a subclass. We embed the interface as a field of an implementation.

Thanks for the detailed explanation! I am one step closer to understanding the standard library now. Also your code snippet does work as expected and does what i want.

Tho I had to change the line:

const stdout_w: *Io.Writer = &stdout_reader.interface; // access stdout_writer's interface by pointer

to

const stdout_w: *Io.Writer = &stdout_writer.interface; // access stdout_writer's interface by pointer