Quasi No-Allocation TCP Stream Duplication in Zig

Showcase
1

Hi folks! You may have seen me asking question around here with regards to std.posix socket connections, or maybe you saw this issue… well, I’m happy to announce my humble foray into the world of Zig: A no-allocation (kind-of) TCP Stream Duplication Program.

What does it do?

a single threaded, asynchronous, quasi no-allocation tcp stream replication service made for implementing semi-distributed systems from monolithic-ly designed applications - the opposite of a load balancer.

Will bind a socket to a specified adddress (using the replicator -p <port> <address to listen on> command).

  • If the address that is returned from posix.accept is found in the src/config.zig input, or output array - it will be categorized as such.
  • An input’s messages will be duplicated to all outputs. If it’s not either an input or output, it will be ignored and the connection will be closed.
  • (somewhat) safely handles disconnecting sockets, and sigint linux signal to attempt a graceful shutdown.
    • I need to make better use of Zig’s errdefer, need some pointers on how to deal with defer vs. errdefer.

Inspiration

I work in DevOps and write some tools in Go. We had a case where we needed to ensure data integrity so we wrote a small proof of concept where we duplicated requests and did some internal logging. That slowly grew into a more mature software. I was interested in rewriting a basic subset of it’s functionality in Zig for fun, and learned a lot.

Misc. Notes

  • I need to write some tests and do some benchmarks… but it works for the basic case of “I have an input, I have outputs, I need the input to write to all the outputs without redesigning the app receiving the data”.
  • Since I take advantage of known inputs and outputs at compile-time, I am able to have all my allocation done with FixedBufferAllocator.
    • Except for the clap parameters stuff, which I maybe could’ve done like that as well, but was out of scope for now.
  • My first work with both sockets, and epoll! Very eye opening.

Future Work

  • Might continue down this rabbit hole, but one thing that caught my eye while working on this was the following idea: What if we fleshed out data structures like std.ComptimeStringMap to allow for cases where the capacity/max_size is comptime known and can be backed by a (global?) array, but allow for runtime puts, gets, etc?
    • It seems you can already kind of do this if your struct is composed of non-variable length items (i.e., for all instances of type @sizeOf(type) will always be same?), since you just have to take into account the meta data required for each item.
    • Not saying this is a good idea, and in fact my intuition says that if you know enough at comptime to determine the max size or capacity then there probably isn’t a need what I just described… I don’t know, something I still need to think about.

Sorry for the wall of text :slightly_smiling_face:. It’s always a big ask, but I would appreciate any feedback on my code. Especially when it comes to the way I did the event_loop, and how to better use errdefer’s.

Thanks!

3 Likes
2

In the Zig Build section, instead of “zig build” I suggest “zig build -p prefix”.

As an example, on Linux: zig build -p ~/.local.

Additionally, in the Run the Service, I suggest removing " The binary is located in the zig-out/bin directory. You can move this to /bin or anywhere else in the PATH that you’d like." .

1 Like