Hi everyone, I’m currently developing an interesting library called z3: S3-compatible storage.
This library is currently in its early stages. Essentially, it transforms zs3’s original sequential processing of S3 requests into multi-threaded concurrent processing. Therefore, most of the processing code has been modified from zs3. First of all, I’d like to thank zs3 for its help in quickly getting started with the implementation of the S3 API.
The most interesting part of this library is its concurrency design, which I will explain in detail.
Every S3 request is split into two halves, A and B:
- A (Server): lightweight operations that need consistency guarantees — access key lookups, global counter increments, metrics updates, log writes. All of A runs sequentially in a single thread.
- B (Client): heavy disk I/O and CPU work — HTTP header parsing, SigV4 HMAC computation, file reads/writes, request routing, response serialization. All of B runs concurrently across a pool of zio coroutines.
Under load this forms a “set of A” and a “set of B”. Serializing A and parallelising B yields the best throughput.
To achieve this, polyrole describes the full lifecycle of an S3 request as a typed state machine. The A set executes in one OS thread; the B set executes across zio’s coroutine pool.
Communication between A and B:
- B → A: a bounded
MsgChannelqueue (capacity 1000) - A → B: a
WaitMsgstruct written directly into B’sClientContext, signalled viaResetEvent
The full A–B protocol is the state machine visualised:
This pattern mirrors the Erlang client-server model: the server (A) is passive, the client (B) sends messages and waits for replies.
Pointer-based communication. A and B pass *ClientContext pointers, so the per-message cost is essentially one ResetEvent syscall — no copies, no serialisation. Polyrole guarantees at compile-time that A never accesses B’s pointer after B has exited, making this safe without runtime checks.
Currently, the client-side S3 processing code fully utilizes zs3 code, which enables the entire architecture to run. The next step is to gradually modify this part of the code to provide full S3 semantic support.
I developed polyrole (originally named polysession, then renamed troupe, and then renamed polyrole again). Initially designed for writing communication protocols, it was later discovered to be useful for writing multithreaded code. Now, it’s shown that, as in this example, collections of clients and servers can operate in different ways. This is very similar to the behavior of Erlang OTP.
Below are some reflections from the development process:
- Currently, I’m using Zio, and it works great! The reason I’m not using the standard library’s fiber is that: the standard library’s coroutines don’t have a complete network implementation, and it seems that recent development has stalled.
- The
Writerrequires asendfile, but there isn’t a good implementation yet. I’m hoping for a zero-copy function similar to Linux’ssendfile(2), but that’s not currently implemented. - In this architecture, communication between the server and the client via mpsc (currently using the channel provided by zio) should not be a bottleneck.
