I want to send bigger chunks of data from a local machine to a storage system using std.http methods. Because the data can be up to some tens or even hundred GiB, it would be nice to track the progress so the user can receive some feedback how long it might take.
But since I’m not very experienced with low-level network stuff, I’m hoping for some tipps.
Whats a good way to accomplish that with std-lib methods? I’ve read through some implementations like std.http.BodyWriter, std.http.Client.Request/Response etc. But since I’m just not very familiar with the underlying concepts, I’m a bit lost. Furthermore, I’m locked in to std.http for now, so custom http libs implemented by other users are no option at the moment.
I’m thankful for any hint; and be it only where to look further.
Hello, i might be out of touch, but std had HTTP/1 implementation.
If you are actually transfering in range of 100 GiB, you might want to consider HTTP/2.
At that point, some dependency will be required.
Thanks for the hint. I don’t know the real size ranges now. But they definitely can be in the multi-GiB range. However, the storage system uses HTTP/1.1, which should be fine with std.http if I’m not wrong.
If you are sending data using the HTTP client, you need to give it a reader for the body, no? So you implement a custom reader that forwards all vtable functions to the original one, but tracks progress.
Ok, that sounds like a good solution, thank you. Does anyone knows of an example/short tutorial how to implement such a custom reader which forwards the vtable functions. While this might be easy/trivial for many programmers, thats what I meant with “concepts” I’m unfamiliar with (while I can use Zig for my own stuff even at work, Im still just a hobby programmer).
Check the implementation of std.Io.net.Stream.Reader (and Writer) It is pretty easy to understand how to implement your own reader or writer from that. You need to implement only one method for each - the one which have no default implementation in the vtable, but since you will probably just be wrapping the Stream variants, you might want to wrap both of the methods Stream implements.
Hi, thanks you for the hint. I’v read through the source code and it seems be doable. However, I’m not really sure how/where to track the data progress and, first of all, how to make the Connection of std.http.Client use my custom implementation. The field Connection.stream_reader expects type std.Io.net.Stream.Reader, how can I put a custom implementation here?
Its absolutely possible that I’m missing something very obvious, because, as stated, I’m not very familiar with this basic concepts of programming languages (vtable’s etc). Unfortunately, therefore its not easy to understand for me (for now, I hope). Thus, I’m happy for any additional info how to accomplish that. But of course, no one should feel urged to do so. Its not the end of the world if the progress won’t get tracked
It seems that I misremember how the API looks like and it does not actually accept a reader for the body. This is actually better for you, you get a writer, so you can track whatever you are writing on your end.
req.transfer_encoding = .{ .content_length = total_len };
var buffer: [4096]u8 = undefined;
var body = try req.sendBody(&buffer);
while (...) |chunk| {
body.writer.writeAll(chunk) catch |err| switch (err) { // recover error.Canceled }
// track progress ...
}
try body.end();
Sorry for the delayed answer. Have been busy at my job.
Right now, the http client is used within a lib. The code for executing the request is the following (very simplified):
// at main.zig create the client and pass it to the underlying library:
var http_client: std.http.Client = .{
.allocator = gpa,
.io = io,
};
// Inside the library, after some preprocessing of other information, the body is sent as
// follows:
http_req.transfer_encoding = .{ .content_length = data.content_length };
var body_writer = http_req.sendBody(&.{}) catch return error.SendFailed;
body_writer.writer.writeAll(data.body) catch return error.WriteFailed;
body_writer.end() catch return error.WriteFailed;
http_req.connection.?.flush() catch return error.FlushFailed
If I understand your example code correctly, I just need to use a custom buffer instead of sendBody(&.{}) and then track the processed chunks inside the while-loop which replaces the body_writer.writer.writeAll(data.body) single line, sending the value “async” to some kind of shared variable. That would mean I have to edit the lib code, but thats no big deal. What would be the correct while (<condition>) in such a case?
Wait, you have the entire multi-GiB file allocated into a single buffer?
The only change then would be to keep what you have, including the empty writer buffer, but instead of writeAll(data.body), you split that into e.g. 1MiB chunks, and track progress after each chunk. Simply have offset as a var, and loop until offset is above the total size.
You are also ignring error.Canceled by those catches, but you might not care about that.
No, I don’t have large files in a single buffer. If the file is larger than e.g. 50MiB, it would be send in multiple requests and merged together on the storage backend again.
Sorry, I missed to add this info to the question. But some users might have slow internet connections and also want to track progress for files smaller than 50MiB or more detailed for larger files than only counting the large chunks (I hope you get what I mean, my English is not the best, especially when typed fast…)
Looks very elaborated, thank you very much. I’m learning a lot here.
But where/how would I “insert” your CountingReader into the http.Client? As value for std.http.Client.reader which is a std.http.Reader type? Thats where I’m very unsure, so many Reader’s in the whole namespace
You would want to wrap the Writer, not the Reader, but the implementation would be very similiar to the Reader provided by xeubie. Then you would probably change this code to something like
var body_writer = http_req.sendBody(&.{}) catch return error.SendFailed;
var writer_counter: CountingWriter = .init(&body_writer.writer, &.{});
writer_counter.interface.writeAll(data.body) catch return error.WriteFailed;
You could even keep the writer_counter and just switch the underlying writer, so that it keeps the count, eg.
var writer_counter: CountingWriter = .init(undefined, &.{});
while (chunks.next()) |*data| {
var body_writer = http_req.sendBody(&.{}) catch return error.SendFailed;
writer_counter.setWriter(&body_writer.writer);
writer_counter.interface.writeAll(data.body) catch return error.WriteFailed;
But of course, it depends how you will be accessing the count, if it will be different thread/fiber then this is fine, but if you want it on the same thread then you would probably need to run additional logic on each write - that could even be implemented in the CountingWriter
So far, this works without problems. Of course, its a very simplified version which can’t stand like this for production code. Nevertheless, it still beats my implementation using the Rust AWS SDK performance-wise (which ofc is not much conntected to progress tracking, but still makes me happy)
@psznm would appreciate it very much if you could explain why to wrap the writer and not the reader as suggestes by @xeubie . This would help me understand those things better. For now, I did not get a cutom writer to work…
Well, because you are writing, not reading. Wrapping reader wouldn’t be of much use to you, since there is no reader to wrap.
If you wrapped the writer and had the debug print logic in it, you could avoid manually splitting the chunks yourself, and have the informational logic separated. The wrapper would cleanly only do informational logic+wrapping, the Io.Reader interface would do the splitting, and this little code slice would just be clean .writeAll( call on the wrapper.
(for now, I hope). Thus, I’m happy for any additional info how to accomplish that. But of course, no one should feel urged to do so. Its not the end of the world if the progress won’t get tracked 
