New Io.Writer use for output redirection

Help
,
1

I have a program that outputs variable size texts to the console. I specifically use std.fs.File.stdout(() for this. However when I redirect the output e.g.

./myprog  > output.txt

the output.txt is incomplete. I made a global reference to a single std.fs.File as received from stdout, and in using that single stdout it does not fix the problem.

After looking around, I am made to believe that I need a single writer, but with the new Zig Io.Writer this is quite a bit more problematic, as its pattern forces me to create smaller writers where and how I need what buffer size to pre-allocate. (the takeaway being that both multiple calls to stdout() as well as stdout.writer reopens the file and creates havoc with the output)

I really hope the answer is not to once off pre-allocate a large enough buffer to hold all output and use the same writer throughout?

What am I doing wrong, or is there another Zig ensuring that output redirection works.

The C equivalent that works is as simple as fprintf(stdout, “whatever”)…

2

Can you elaborate why a single writer is a problem for you?

3

Did you call flush() after you are done writing?

4

In order to use a single writer, I have to allocate and manage a variable sized buffer as you cannot have a writer without a backing buffer. Having a single writer means having a single buffer. Or am I missing something?

5

With it being multiple writers that go to the same file, yes I do call flush multiple times. I thought this had to be done as the flush is to get the buffer to the underlying file via the writer.

6

You can just choose the buffer size (in most cases). If you choose a buffer size of e.g. 4096 bytes, once the buffer is full, its contents are written to the file.

The cases where you can’t choose the buffer size freely are things, like compression, which require a minimum buffer size, but writing to stdout should work with any buffer size.

7

I did not know that… I always take extreme care to whatever I put in the buffer to be able to fit in the buffer. What you say feels strange - but I will give it a go and test it. Thanks

8

This is the implementation of Writer.write:

pub fn write(w: *Writer, bytes: []const u8) Error!usize {
    if (w.end + bytes.len <= w.buffer.len) {
        @branchHint(.likely);
        @memcpy(w.buffer[w.end..][0..bytes.len], bytes);
        w.end += bytes.len;
        return bytes.len;
    }
    return w.vtable.drain(w, &.{bytes}, 1);
}

If there is space for new bytes in the buffer, then they are copied into the buffer. In other cases, the buffer as well as the new bytes are “drained”, which in the case of a file writer means, that they are written to a file.

Writer.write is not the only method of Writer, which writes, but the other methods do similar things.

9

@rpkak - big thanks. Yes it is working! So what is the rule? Does this code mean that the buffer should at least be the size of the biggest single write? Or is the buffer simply a cache only?

10

There is not really one rule. If less performance in writes are not a problem to you, it doesn’t really matter. If performance in writes is a big problem to you, you should just try out and benchmark. Also Andrew wrote some recommendations here.

11

Thanks, I read Andrew’s post as well. Thanks again for the quick response. I will keep on doing a few tests, for there are a few things that still bother me.

12

FYI based on what you’ve said you don’t seem to be aware you can use a zero length buffer &.{}, this causes all writes/reads to be unbuffered and removes the need to flush. It is ofc less performant if you do more than one read/write.

1 Like
13

No I definitely did not know that. I honestly thought that buffer, that seemed mandatory, is a staging area where there at least had to be sufficient space for a single (albeit partial) write; again a prejudice from other languages and toolsets (e.g. snprintf).

It is simply a cache in other words to be more specific than buffer. It makes perfect sense now, thanks.

The pattern for the best/correct buffer is still applicable to other things (like formatting into strings/arrays etc)… which also contributed to me keeping to the same pattern.

14

@vulpesx , so stuck on a size I tested with a buffer of size 1, didn’t even think to test with an empty buffer - but thanks again. Worked with a size of 1 and confident you are correct that it will work with empty buffer too.

15

A non-zero length buffer is required for some functions, mostly take and peek functions, but I think some others too.

They document their requirements and also assert the buffer size so you will know if it’s a problem

16

Understood thanks. At least my understanding has better foundation now; and can grow it from there. Sadly I have spent some time on carefully contemplating possible size in places where it simply did not matter :slight_smile: