Understanding `std.compress.zstd`

Help
1

The documentation for std.compress.zstd is confusing.

I tried going through the code but it’s beyond my knowledge. is there an proper community made documentation or explanation / example would be grateful.

Thank you.

2

If you have a slice of compressed data and you just want a heap-allocated slice of the decompressed version, then the logic in testDecompress will work:

var out: std.Io.Writer.Allocating = .init(allocator);
defer out.deinit();

var in: std.Io.Reader = .fixed(compressed);
var zstd_stream: std.compress.zstd.Decompress = .init(&in, &.{}, .{});
_ = try zstd_stream.reader.streamRemaining(&out.writer);

// out.written() contains the result

However, it’s really important to note that this example provides a zero-length buffer to the zstd.Decompress instance, which puts it in “direct” mode (more context), which only works when:

  • You only call functions that take a Writer (relevant issue), and
  • The output buffer length is >= Options.window_len + std.compress.zstd.block_size_max, or you are using Writer.Allocating which will resize to be large enough as-needed

If you don’t control how the zstd.Decompress will be used (i.e. you’re passing &decompress.reader to a function that takes a *std.Io.Reader) or you know you’ll be calling functions that rely on the reader’s buffer being non-zero-length (peek, take, many others), then you need to provide a buffer to zstd.Decompress that has a length >= Options.window_len + std.compress.zstd.block_size_max. This will put it in “indirect mode” which means it will first write to the reader’s buffer, which may or may not be redundant work depending on your usage.

To that end, I think something like this as an addition to the zstd.Decompress.init doc comment might be helpful:

/// `buffer` fulfilling the requirements supersedes/obsoletes the requirements on
/// the `Writer.buffer`, but may introduce redundant memcpy calls. This can be useful
/// when you do not know/control the buffer capacity of the `Writer`.

(but it’d also need a heavy warning about the limitations of “direct” mode)

5 Likes
3

Thank you. that was very helpful.

2 Likes