Streaming decompress example

Help
1

So following on my post from a couple of months back (Help with new deflate functionality) I’m having trouble figuring out the other way around, and can’t find any examples for what I want to do.

Most examples of decompression with std.compress.flate.Decompress seem to utilize a fixed buffer for the input. What I’m looking to do instead is to be able to stream the input in a push-type model, so that it does not all need to sit in memory.

What I thought I could possibly do, albeit it being awkward, was set up some sort of writer with a buffer that was shared as the input buffer for std.compress.flate.Decompress, where decompression would be handed on drain on the writer side or something (with an appropriate refill/rewind of the input buffer). I’m running into issues with that though, and the general awkwardness of the implementation is making me think there’s got to be a better way that I’m missing.

Any thoughts?

2

PS: I’m guessing the correct way may be to actually write a reader with a stream implementation and not a writer… I am going to attempt to do that, but still would love to hear any suggestions!

3

Figured it out. :slight_smile: This is a pretty simple implementation and might not be the most performant, but it works for my purposes and for demonstration. To that end, this “throttles” a larger buffer of sample compressed data that would likely be able to fit into memory by breaking it up into chunks of 8 bytes. The actual compressed data goes into src. Obviously, for real world purposes, the buffer for the reader would be much larger and the data would be fetched from an outside stream, and possibly being passed directly to the writer, but this implementation does keep things simple and ensures reads are buffered as much as possible.

Any critique is welcome.

// Assume compressed data in `compressed`.
var decompressed_buf: [64]u8 = undefined;
const decompressed = decomprssed: {
    const InputReader = struct {
        const buffer_size = 8;
                                                                                                               
        src: []const u8,
        pos: usize = 0,
        reader: std.Io.Reader,
                                                                                                               
        fn stream(r: *std.Io.Reader, w: *std.Io.Writer, limit: std.Io.Limit) std.Io.Reader.StreamError!usize {
            const self: *@This() = @fieldParentPtr("reader", r);
            _ = w;
            _ = limit;
            @memmove(r.buffer[0 .. r.end - r.seek], r.buffer[r.seek..r.end]);
            r.end = r.end - r.seek;
            r.seek = 0;
                                                                                                               
            const src_end = @min(self.pos + (buffer_size - r.end), self.src.len);
            const actual_len = src_end - self.pos;
            @memcpy(r.buffer[r.end .. r.end + actual_len], self.src[self.pos..src_end]);
            r.end += actual_len;
            self.pos += actual_len;
            return 0;
        }
    };
                                                                                                               
    var input_buf: [InputReader.buffer_size]u8 = undefined;
    var input: InputReader = .{
        .src = compressed,
        .reader = .{
            .buffer = &input_buf,
            .vtable = &.{
                .stream = InputReader.stream,
            },
            .seek = 0,
            .end = 0,
        },
    };
                                                                                                               
    var zlib_buffer: [std.compress.flate.max_window_len]u8 = undefined;
    var zlib_stream: std.compress.flate.Decompress = .init(
        &input.reader,
        .zlib,
        &zlib_buffer,
    );
                                                                                                               
    const decompressed_len = try zlib_stream.reader.readSliceShort(&decompressed_buf);
    break :decomprssed decompressed_buf[0..decompressed_len];
};
4

I am confused, you asked for how to use a non-fixed buffer to avoid having the whole data in memory.

Then proceed to implement your own reader that reads from a fixed buffer..?

I am also not sure why you think std.compress.flate.Decompress uses a fixed input buffer? You literally pass it a reader in your own code, the buffer you pass it in init is for its own decompress.reader.

5

src is just for demonstration purposes. I mention this, but it may not have been clear. I’m saying that in a real-world scenario that data would be coming from somewhere else; in my case, PNG data, so it needs to be fetched chunk-by-chunk and streamed in, and a scanline reader needs to be able to pull whole decompressed scanlines from the stream.

I just mean that the tests and basic examples I’ve seen so far use fixed input buffers. Not that this would be how you would want to do it normally, which is what I was trying to figure out.

6

you’d just pass the reader for the compressed data to the decompressor. Assuming you have one.

thats why the decompressor has requirements on both the readers buffer and the other buffer you pass it:

/// `input` buffer is asserted to be at least 10 bytes, or EOF before then.
///
/// If `buffer` is provided then asserted to have `flate.max_window_len`
/// capacity.
pub fn init(input: *Reader, container: Container, buffer: []u8) Decompress {

for future reference just use a std.Io.Reader.fixed, the intent will be much more clear, and it will be more efficient than what you implemented.