Do I misuse or don't understand Io.Writer?

Help
1

I have a function that collects, compress, writes data to a buffer.

    const bound = try encoding.compressBound(self.uncompressedMetaindexBuf.items.len);
    try self.metaindexBuf.ensureUnusedCapacity(bound);
    try self.metaindexBuf.writer.ensureUnusedCapacity(bound);
    const n = try encoding.compressAuto(
        self.metaindexBuf.unusedCapacitySlice(),
        self.uncompressedMetaindexBuf.items,
    );
    self.metaindexBuf.items.len += n;

Here I compress, preallocate at once and write to the destination the content.

Now I need a file as a destination, so it feels correct to use Writer, but

  1. Writer doesn’t support “close” api, so it means I have to hold a link to a file to close it? so it means I hold them nullable, and if I have 3 destination implementations (e.g. ArrayList, File and S3) then I might have 3 fields per buffer to hold different deinit, and I have 4 of them, so it’s 12 fields in total only for deinit?
  2. no direct access to a file buffer, therefore I need to allocate an intermediate buffer to write a compressed content and then write the buffer to a file/destination array, right? can I trick somehow the implementation to use a file’s buffer, in streaming mode write compression and avoid intermediate buffer allocation?
2

Just take an *Io.Writer, the caller can deal with what writer they are using and closing the file if they are using one.

You can require the writer have a minimum buffer size and use its buffer.
But that might not be the best solution, depending on what you’re doing with it. Ideally, you wouldn’t need an intermediary buffer and could just write to the writer.

1 Like
3

creating the files in the caller is overwhelming, it duplicates quite a lot of code across the codebase, but yeah, it’s the only option at the moment.

4

sounds to me like you just need to write another function to deduplicate the filesystem stuff?

5

That’s what I expect from interfaces, yes. But apparently I need to implement my own writer for both

6

I am very confused, what else do you think you need to implement your own writer for?

Is it the file? That is not the case, you just use the existing implementation.

I also dont think opening and closing a file is worth concern of code duplication. If there is other things to get the file in the first place, or things to do with the file before you write your compressed data to it, those are separate things which you can probably encapsulate in a function(s).

but the opening and closing of a file, namely the defer file.close cannot be encapsulated in another function.

7

Frankly the fact that it can’t close your file is an amazing feature. As anyone who has ever struggled with libraries returning fd to the kernel underneath them will surely attest…

1 Like
8

…why would it have knowledge of when you want to close your file? Io.Writer is far more generic than that.

2 Likes
9

Elaborating a little: imagine using a library function that you don’t control that accepts a pointer to a writer. You know that if you open a file, make a writer on it, and pass that to the library function, then your file is still open afterwards. If writer interface would implement a close capability, you’d lose that guarantee (because presumably the standard file writer implementation would implement it the obvious way; you could always roll your own vtable to control the close - but you’d still have to worry about how to behave if the library throws a call to close in there).

1 Like
10

I appreciate this explanation, I was about to start googling what you had said previously but this makes perfect sense and is a +1 design decision.