Stdout file writer does not seem to respect O_APPEND

Help
,
1

Hi!

I am currently trying to write a program that prints its results to the terminal via stdout. As part of this, I am trying to have it append to a file each time it runs by executing ./path/to/exe >> output.txt.

However, when I do this, it appends the output to the top of the file, overwriting any existing contents instead of appending to the end of the file like it is supposed to.

I’m happy to provide code, but this bug is also present in the default (0.16.0) zig init project (I copied my stdout printing code from there) which I imagine is an easier minimal example of the issue.

I have verified that my shell is working correctly (echo hello world >> test.txt works as intended). I am running ZSH on macOS with Apple Silicon (aarch64).

Thank you in advance for your help!

2

Hello, i fixed it by seeking in the writer.

const handle = std.Io.Dir.cwd().createFile(ctx.io, "log.txt", .{ .read = false, .truncate = false }) catch return;
defer handle.close(ctx.io);

var logger_buffer: [1024]u8 = undefined;
var logger_writer: std.Io.File.Writer = .init(handle, ctx.io, &logger_buffer);

if (handle.length(ctx.io)) |len| {
   logger_writer.seekTo(len) catch {};
} else |_| {}

const writer = &logger_writer.interface;
defer writer.flush() catch {}; // famous "dont forget to flush"

// use writer further ...

I am not saying this is the perfect fit all solution, this code handles errors desireably for my solution, however you might want to change that.

If anyone knows a better / cleaner way, please let me know too.

Robert :blush:

3

The issue is that the file reader/writer use positional syscalls by default.

@Bobvan gave one solution: changing its position to the end of the file.

Another solution is changing the mode to use streaming syscalls which should respect the append flag. You can do this either by creating it with the *Streaming version of the writer/reader functions. Or by setting the mode field to .streaming.

It is important to know my alternative solution means using a global file position, managed by the Os, which is probably what you are used. Just be aware as they have different behaviour when you have multiple readers/writers.

1 Like
4

Not a bug. As @vulpesx alluded to, you’re in positional mode, but need streaming mode. The zig init version opens stdout positionally, whereas you want something like

pub fn main(init: std.process.Init) !void {
    const io = init.io;
    var stdout_buffer: [1024]u8 = undefined;

    var stdout_file_writer: Io.File.Writer = .initStreaming(.stdout(), io, &stdout_buffer);
    // --------------------------------------^

    const stdout_writer = &stdout_file_writer.interface;
    try stdout_writer.writeAll("append this\n");
    try stdout_writer.flush();
}

and ./path/to/exe >> output.txt works as expected.

5

Ah, I should also mention that they will automatically switch to streaming if the file is not seekble.

The downside is that wastes a syscall, though you will often call in loops, or use higher level functions that themselves loop, so you usually don’t have to do much (if anything) to account for this.

6

Isn’t it a bug in zig init? You should always use streaming mode for stdout.

7

Thank you all for your help!! What are the advantages of positional write over streaming?

8

See my above comment; it is not a bug, but it is less than optimal.

9

I think using handle is incorrect, since it is a low level abstraction from the OS. Why not using file?

10

Its data type is std.Io.File, i just named it handle, since std.Io.File in zig is “file handle” with flags.

I concede i possibly could have named it better, however i am used to naming my “handle with flags” as handle.

Robert :blush:

11

Hmm, this sure is surprising behaviour. It does make sense that it happens, given that std.Io uses positional syscalls by default, but it is still surprising, especially for the shell redirection case.

I wonder whether it would be worth it for std.Io.File.Writer to default to the file descriptor’s current seek position (when available) upon construction :thinking:

1 Like
12

It is more friendly with threads. Imagine you have many threads that access the same file. The position would be all over the place. You would do it with some kind of database file not tty or shell redirection.

I guess ZSH or macOS does something different from Bash and Linux. I could not reproduce the bug. I guess bash magically forces the reader to switch to streaming mode while zsh does not.

1 Like