Track transferred bytes/progress of longer PUT request with std.http

Help
,
21

You got the wrong userdata. Basically you are passing pointer to your own Writer interface, to the underlying writer which expects pointer to its own Writer interface. Should be

        const n = try self.inner.vtable.drain(self.inner, data, splat);

Otherwise looking good. Took me a while to spot the issue.

2 Likes
22

Hey, thank you very much. It totally makes sense, but is hard to get while reading through the source code.

It now works with this code:

pub const CountingWriter = struct {
    inner: *std.Io.Writer,
    pos: u64 = 0,
    interface: std.Io.Writer,

    pub fn init(inner: *std.Io.Writer, buffer: []u8) CountingWriter {
        return .{
            .inner = inner,
            .interface = .{
                .vtable = &.{
                    .drain = drainFn,
                },
                .buffer = buffer,
                .end = 0,
            },
        };
    }

    fn drainFn(w: *std.Io.Writer, data: []const []const u8, splat: usize) std.Io.Writer.Error!usize {
        const self: *CountingWriter = @fieldParentPtr("interface", w);
        const n = try self.inner.vtable.drain(self.inner, data, splat);
        self.pos += n;
        std.debug.print("{d}\n", .{self.logicalPos()});
        return n;
    }

    pub fn logicalPos(self: *const CountingWriter) u64 {
        return self.pos;
    }
};

Thanks @psznm , @xeubie and @lalinsky for all the good examples and advices. I’ll mark this as solution, because its working code; so anyone who is looking to solve a similar task finds it. But of course, I would not have worked it out (this fast or maybe ever?) without help of the named users!!

1 Like
23

I took the inspirations and tipps from this thread and implemented wrapping Reader and Writer for the http requests of the S3 lib z3. I hope this is might help somebody who has a similar task to solve. And of course thanks again to anybody who answered in this thread!

1 Like
24

Did this work out well for you? I found myself in need of something similiar, I also implemented it by calling vtable of underlying reader/writer. Specifically net.Stream.Reader and net.Stream.Writer. And found the counts I was getting very suspicious.

I found that Reader.readVec will hide bytes from you (those that it stores into buffer internally), and its stream also calls that. I found that finding the actual bytes read can only be done by actually copying the underlying implementation, or by doing counts relying on precise knowledge of underlying implementation. So with reader this works best if the underlying reader has no buffer.

For writer I found similiar thing, if I want to have a buffer on the wrapping writer (some things require it) and drain correctly, I need to pass the buffer to the underlying writer. Then it will also not report the data drained from buffer so I need to account for that when counting.

25

Hi, yes after reworking the ideas found here a little bit we implemented a tracking reader/writer implementation in the z3 s3 library:

https://codeberg.org/fellowtraveler/z3/src/commit/388100be234b838778fc974b645596b1616c2762/src/http/request.zig#L461-L625

The wrappers are used in all executeRequest... functions in the same file

You can find a real world example in the repo of my s3 CLI client. E.g. here: https://codeberg.org/lukeflo/z3cli/src/commit/c66b0913fedddadf27f1c8856f0eab674497ace3/src/subcommands/Put.zig#L436

Edit: it doesn’t wrap net.Stream.Writer, but http.BodyWriter

But I might confuse some thing since I write from smartphone

26

I see. That explains things. I had to jump through some additional hoops to get correct behavior when wrapping Io.Reader and Io.Writer from net.Stream.Reader and net.Stream.Writer

EDIT: if anybody comes here looking for answers (I think this should work correctly for any underlying reader/write as long as the underlying reader/writer does not have to have a buffer):

Writer:

        fn drain(w: *Io.Writer, data: []const []const u8, splat: usize) Io.Writer.Error!usize {
            const self: *Writer = @alignCast(@fieldParentPtr("interface", w));
            var out = self.output;
            out.buffer = w.buffer;
            out.end = w.end;
            const end_pre = w.end;
            const data_written = try self.output.vtable.drain(out, data, splat);
            w.end = out.end;
            const actually_written = data_written + end_pre - w.end;
            self.stat.operated(self.io, actually_written);
            return data_written;
        }

Reader:

        fn stream(r: *Io.Reader, w: *Io.Writer, limit: Io.Limit) Io.Reader.StreamError!usize {
            const self: *Reader = @alignCast(@fieldParentPtr("interface", r));
            assert(self.input.buffer.len == 0); // Would not report actual bytes read
            const read = try self.input.vtable.stream(self.input, w, limit);
            self.stat.operated(self.io, read);
            return read;
        }
1 Like