Hi, for my game, I’m trying to implement an efficient asset streaming pipelines, that leverages the Io interface, for concurrent, mmaped positional reads, of multiple files. I’m close to finishing, and I’d like the final output to be basically a list of files, with their respective readers interface, that would under the hood hydrate the available data, by consuming the FileChuncks of that file (a mini struct with an offset/len, built ahead of time). The thing that I want to know, is what is the minimal set of functions to implement a Reader ?
If you are using mmap, just use fixed buffer reader. Why would you want anything more compex?
1 Like
Hi, I can’t use FixedBufferReader, because the way I want to read, is concurrent across big files, to do that for each files, i build a list of chuncks which are a list of struct with an offset/len. The reason i do that is such that I can hash the file in parallel using Blake3, whilst reading them, so for the code that just uses the reader, it gets the bytes, but underneath, in the reader I want to have a map of the files_path / with their hash, to ensure the integrity of the assets.
You are losing the benefit of mmap then, because you will have to copy data from the mapped memory to the reader buffer.
You need to implement stream, readVec and maybe discard. Look at the the decompressors in the standard library, e.g. the flate one, as a fairy simple example.
Maybe you could play some tricks with the internal buffer and rebasing, if you want to make it zero copy.
https://codeberg.org/ziglang/zig/src/branch/master/lib/std/compress/flate/Decompress.zig
2 Likes
Yeah I’m not sure my idea will work great, but best way to know is to test it out 
Thank you for the help
The only one that is strictly needed is stream. There are defaults available for the other ones which you can use.
I’ve done something similar before with a zero copy mmap approach and basically settled on using an empty buffer in the interface. This way everything just operates directly on the mapped file. You then may have to change some of the other vtable functions and likely disallow rebasing as it’s not really applicable. See this post which is somewhat similar.
But if the memory and file sizes allow it I would likely just let the operating system map it in and do madvise with MADV_DONTNEED or just munmap it entirely. This can save some headaches.
2 Likes
Thanks the idea of the reader is less about zero copy, and more about Concurrent positional reads, basically instead of being read sequentially, the file is read concurrently multiple chunks at the time, each chunk gets blake3, under the hood, the reason is that in my game, many assets are going to be on the same files, things like sprite sheet for example, so instead of loading/reading all the file, i can just load a given chunk, compare it’s hash and ensure it’s correct. Which might be completely overkill, but it’s fun to tinker with so
Well After testing, The results aren’t really promising, after testing my idea it turns out bad, higher latency, and 40% less throughput (25gb/s sequential vs my multi threaded 15 gb/s, whilst consuming more memory too) than the dumb sequential read approach, was interesting nonetheless.
Consider mmap + MADV_SEQUENTIAL
1 Like
I did but with the overhead of threads, it’s actually dominating even on large files, like 8gb. Having said that I’m on a DGX spark at the moment, and the thing as stupid disks/ram speed, with sequential i can even get 40gb/s if the file is hot. But even if it was faster it was a bit awkward. I’m testing other ideas rn thanks tho
1 Like