Need help understand readfileAlloc()

Help
, ,
1

So basically i have a file with the size of 1200 in bytes.

 ❯ stat -c %s input.txt
1200

I tried initialize buffer with a size of 1200 bytes and always get error OutOfMemory() when running the code below? I tried using binary search method to find the find the smallest buffer size that it doesn’t cause OOM and its 1928?

Can someone explain how this happen? Am i misunderstand something, or am I missing something obvious?

const std = @import("std");
const Io = std.Io;

pub fn main(init: std.process.Init) !void {
    const io = init.io;
    var buffer: [1928]u8 = undefined;
    var fba = std.heap.FixedBufferAllocator.init(&buffer);
    const allocator = fba.allocator();
    
    const data = try Io.Dir.cwd().readFileAlloc(
        io,
        "input.txt",
        allocator,
        .unlimited,
    );
    defer allocator.free(data);
    std.debug.print("Read {} bytes\n", .{data.len});
    //Output: Read 1200 bytes
}
2

The specifics of why this is the case are going to be visible in the FBA and file reader implementations, so it’s worth looking in there. It could be allocation headers, alignment, maybe some fragmentation caused by readFileAlloc needing intermediate allocations? (the following post actually explains it better!)

Regardless it’s probably more useful to say that this isn’t really a good way to use readFileAlloc - just give it a normal gpa.

Either that or use a fixed writer to stream the bytes into a buffer, or similar.

1 Like
3

The reason is simple, readFileAlloc() does not querry the file size when reading.
It just initializes arraylist and then streams the data into it. However this arraylist grows independant to the file size (it will overgrow it).

Here is the growing logic for arraylist:

/// Called when memory growth is necessary. Returns a capacity larger than
/// minimum that grows super-linearly.
pub fn growCapacity(minimum: usize) usize {
    if (@sizeOf(T) == 0) return math.maxInt(usize);
    const init_capacity: comptime_int = @max(1, std.atomic.cache_line / @sizeOf(T));
    return minimum +| (minimum / 2 + init_capacity);
}

If you need to have the allocation consume exactly the file size memory, querry the file size, allocate buffer of that size and then read into it.

If you need anything more explained, feel free to ask.
Robert :blush:

4 Likes
4

I know, im just new to zig and trying to do everything with allocator (honestly im still not fully understand it) so i just tried to play around with allocator as much as i could. Even after i read the official docs of zig (Memory: How to choose an allocator), i dont even know which allocator should i choose​:face_with_bags_under_eyes:

1 Like
5

Thanks man, so that’s why. Much appreciated

1 Like
6

ㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤ

1 Like
7

Maybe that’s an optimisation worth a PR.

After all I don’t really see the point of not doing that since that way one could read in one read syscall instead of multiple smaller ones.

8

So, as I understand it, the link that @anon47052109 provided contains a section on using readFile where you can allocate a single buffer that’s the same size as the file itself. I think the point is that using the readFileAlloc uses an ArrayList which needs more memory than just the file contents since it has some additional features compared to a raw buffer. The OOM crash is because there is a FixedBufferAllocator being used which does not have enough memory available for the ArrayList and the actual file contents

9

ㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤ

10

I think you misunderstood @KilianHanich’s reply: the point is that the source code for readFileAlloc could very well (in theory) instead do exactly what @Bobvan describes, not using std.ArrayList at all. “Maybe that’s an optimisation worth a PR” means “Maybe it would be worth changing the function to do this.”

1 Like
11

I would guess that the usage of ArrayList is to make reading the file and manipulating the contents easier, compared to a fixed size buffer

12

yes, but also no! When streaming the data, a file reader will redirect to sendFile, making the writer aware it is dealing with a file.

The allocating writer (made from the array list), when doing sendFile does query and allocate the size of the file. However, the allocation logic will @Bobvan showed is still used, so it can, and does in this case, allocate more than the exact size.

This could be changed to more precisely grow when it has a file size, the code to do this already exists… it just isnt used in this code path.

Its just cause the simplest api path to do this happens to use an arraylist.
This api does not, and should not be, manipulating the contents beyond reading and growing.

1 Like