Back to Basics: File Reading with the New IO in Zig

ww520 · September 24, 2025, 1:08am

Zambyte · September 24, 2025, 2:47am

Small note but GeneralPurposeAllocator was renamed to DebugAllocator in 0.14.0

tensorush · September 24, 2025, 4:22am

I’m not sure reader.toss(1) is the right API for discarding the delimiter byte, have you tried using takeDelimiterExclusive?

ww520 · September 24, 2025, 4:37am

Ha. Copying and pasting couple month old code are outdated already. Thanks for the info! I’ve updated the blog with DebugAllocator.

plaukiu · September 24, 2025, 4:46am

careful std.Io.Reader: fix delimiter bugs by mlugg · Pull Request #25169 · ziglang/zig · GitHub

ww520 · September 24, 2025, 4:54am

Good question. The current implementation of takeDelimiterExclusive only looks in the buffered data up to the buffer’s capacity for the delimiter. It won’t look beyond that. See the called peekDelimiterInclusive doing the read; the while loop only ranges over the buffer’s capacity.

To make it work, you have to have a big enough buffer to read in the data with the delimiter. That’s too much assumption for a general purpose read-until-delimiter.

I’ve gone through all variations of the delimiter reading API and settled on the streamDelimiter() with a growing writer buffer.

AndrewKraevskii · September 24, 2025, 3:19pm

In provided snippet:

var reader = &file_reader.interface;

reader variable can be made const since it’s a pointer and not a value. Compiler sadly can’t detect it

ww520 · September 24, 2025, 3:56pm

I remember seeing a compile error: expected type ‘*Io.Reader’, found ‘*const Io.Reader’ at reader.streamDelimiter(…) before and changed it to var. Now that I change it back to const as you suggest and it works. I guess it was the ‘const file_reader’ causing it and it showed up as a compile error on ‘reader’. Thanks for the info.

wjlewis · December 4, 2025, 3:18pm

Thanks for writing and sharing this! It helped me a lot.

If I want to read the entire contents of a file (instead of line-by-line), is this the best approach:

Create an ArrayList(u8) in which to store the contents.
Call appendRemaining on the reader, passing a suitable allocator and the array list.

?

Specifically:

// --snip--
const reader = &file_reader.interface;
var text_buf = std.ArrayList(u8).empty;
// Read the entire file into `text_buf`.
try reader.appendRemaining(alloc, &text_buf, .unlimited);

// Get a slice to the text content. Alternatively, call `toOwnedSlice()`.
const text = text_buf.items;

ww520 · December 4, 2025, 3:53pm

Thanks for the comment. For reading the entire file into memory, using appendRemaining works when loading into an ArrayList. There’re several other ways. The following are some examples.

This is an one-liner, simplest.

github.com/williamw520/misc_zig

read_full1.zig

main

const std = @import("std");

// Read a file fully into a buffer.
pub fn main() !void {
    var gpa = std.heap.DebugAllocator(.{}){};
    defer _ = gpa.deinit();
    const alloc = gpa.allocator();

    var argv = try std.process.argsWithAllocator(alloc);
    defer argv.deinit();
    _ = argv.next();
    const filename = if (argv.next()) |a| std.mem.sliceTo(a, 0) else "test.txt";

    const data = try std.fs.Dir.readFileAlloc(std.fs.cwd(), filename, alloc, .unlimited);
    defer alloc.free(data);

    std.debug.print("{s}\n", .{data});
}

This one let you allocate a buffer first.

github.com/williamw520/misc_zig

read_full2.zig

main

const std = @import("std");

// Read a file fully into a buffer.
pub fn main() !void {
    var gpa = std.heap.DebugAllocator(.{}){};
    defer _ = gpa.deinit();
    const alloc = gpa.allocator();

    var argv = try std.process.argsWithAllocator(alloc);
    defer argv.deinit();
    _ = argv.next();
    const filename = if (argv.next()) |a| std.mem.sliceTo(a, 0) else "test.txt";

    var file = try std.fs.cwd().openFile(filename, .{ .mode = .read_only });
    defer file.close();
    const data = try alloc.alloc(u8, (try file.stat()).size);
    defer alloc.free(data);
    _ = try std.fs.Dir.readFile(std.fs.cwd(), filename, data);

    std.debug.print("{s}\n", .{data});

This file has been truncated. show original

This one is most efficient, least amount of buffer copying.

github.com/williamw520/misc_zig

read_full3.zig

main

const std = @import("std");

// Read a file fully into a buffer.
pub fn main() !void {
    var gpa = std.heap.DebugAllocator(.{}){};
    defer _ = gpa.deinit();
    const alloc = gpa.allocator();

    var argv = try std.process.argsWithAllocator(alloc);
    defer argv.deinit();
    _ = argv.next();
    const filename = if (argv.next()) |a| std.mem.sliceTo(a, 0) else "test.txt";

    var file = try std.fs.cwd().openFile(filename, .{ .mode = .read_only });
    defer file.close();
    const file_size = (try file.stat()).size;
    const data = try alloc.alloc(u8, file_size);
    defer alloc.free(data);
    var file_reader = file.reader(data);
    try file_reader.interface.fill(file_size);

This file has been truncated. show original

This one is for when the file size is unknown, reading as much as possible until the EOF.

github.com/williamw520/misc_zig

read_full4.zig

main

const std = @import("std");

// Read a stream fully into a buffer without knowing the data size.
pub fn main() !void {
    var gpa = std.heap.DebugAllocator(.{}){};
    defer _ = gpa.deinit();
    const alloc = gpa.allocator();

    var argv = try std.process.argsWithAllocator(alloc);
    defer argv.deinit();
    _ = argv.next();
    const filename = if (argv.next()) |a| std.mem.sliceTo(a, 0) else "test.txt";

    var file = try std.fs.cwd().openFile(filename, .{ .mode = .read_only });
    defer file.close();

    var read_buf: [4096]u8 = undefined;
    var file_reader = file.reader(&read_buf);
    const reader = &file_reader.interface;

This file has been truncated. show original

Cheers.

jmctagger · December 4, 2025, 6:16pm

Funny that this should resurface today.

Earlier, stratts advocated for not assigning the interface to a variable at all.

Indeed, in this case, in your post:

    var file_reader: std.fs.File.Reader = file.reader(&read_buf);

    // Pointer to the std.Io.Reader interface to use the generic IO functions.
    const reader = &file_reader.interface;

You create a reader just as in the discussion (and oops) of that thread. I notice that file_reader is used for nothing other than its interface, and that the interface (reader) is used only twice. Obviously, in a real app, it would potentially be used much more, and file_reader might, too, but would it make sense to propagate @stratts proposal, and just

    _ = file_reader.interface.streamDelimiter(&line...

instead, and possibly rename variables for ergonomics?

ww520 · December 4, 2025, 7:17pm

stratts and you have a good point.

vulpesx · December 5, 2025, 12:27am

There should be no buffer copying in half of your solutions.
The reader will first copy data from its buffer to the output buffer/writer if it has any data, then it reads the data directly into the output buffer/writer. In your examples, you don’t do any operation that puts data in the internal buffer, Excluding the example where you only fill the buffer.

Except Writer.Allocating, as that might copy data when growing its buffer
Also except Dir.readFileAlloc as that appends to an array list

ww520 · December 5, 2025, 12:59am

Trace through the call stack of std.fs.Dir.readFileAlloc.

wjlewis · December 5, 2025, 2:12am

Thank you again! That’s much clearer that what I was doing.

ww520 · December 6, 2025, 12:54am

There seem to be quite a bit of questions regarding file I/O. I’ve put together two more blog posts diving into the topic.

Most languages have an one-liner library function to read a text file into lines. The above is a version for Zig. It supports a simple usage like below.

var f = try FileLines.read(alloc, std.fs.cwd(), "test.txt");
for (f.lines()) |line| {
    std.debug.print("line = '{s}'\n", .{line});
}
f.deinit();

mg979 · December 6, 2025, 2:19am

Thank you for this, any idea about how the best way to make partial reads/writes? For example I could stop reading after N lines, then read the rest by setting position with file.seekTo(), but then on write, how can you write only a part of the file, assuming you didn’t read it all and edited a part of it?

npc1054657282 · December 6, 2025, 2:24am

I think streamDelimiter should be helpful.

ww520 · December 6, 2025, 10:50am

std.fs.File.Reader.seekTo() should work with partial reading with jumping to different parts of the file. It will adjust the associated std.Io.Reader interface for the seek operation.

std.fs.File.Writer.seekTo() also should work. It will adjust the associated std.Io.Writer interface for the seek operation. You just seek to a position in the file and write the number of bytes for the partial data.

Also the File.Reader and File.Writer are opened with the .positional mode by default.

mg979 · December 6, 2025, 1:11pm

Thanks, I’ll try with this one.