ZYAMP or do we need Yet Another Messaging Protocol?

Brainstorming
1

Recently, I’ve been thinking: how are we supposed to connect systems written in Zig?

Not everything is a web client, a broker-based system, or a part of some structured RPC flow.

Sometimes, you just want to exchange good old-fashioned messages—without worrying about transport-level details or heavyweight abstractions.

So I present to you ZYAMP - Zig’s Yet Another Messaging Protocol.

It’s an attempt to define a

  • minimalistic
  • peer-to-peer
  • asynchronous
  • message-based
  • transport-agnostic
  • friendly to native Zig protocol.

I’d like to hear your thoughts:

  • Is ZYAMP a meaningful direction?
  • Are there use cases you’ve faced where something like this would help ?
  • Does Zig need a protocol like this
  • Or if existing ones already serve us well enough

Thanks to ChatGPT and Grok for their patience while I was polishing these ideas.

2 Likes
2

IMHO there’s still too many transport layer details in the readme (like “designed for peer-to-peer” - this should be irrelevant for a message protocol), currrently it’s a bit unclear if this is heading for something like protobuf, ZeroMQ, a “better” HTTP or RakNet/ENET.

All those have wildly different use cases and have very different designs for good reason (Protobuf is purely a protocol codec, and doesn’t care about the transport layer, ZeroMQ is nice for communicating between services, HTTP is nice for client/server communication, and RakNet/ENET are good for games but only care about the transport layer but not the message protocol format).

I would go for a pure protobuf replacement (e.g. just the message encoder/decoder), and ideally attempt to use Zig’s comptime reflection features to derive the message format directly from structs - or maybe alternatively go the other way around, describe the message format in a .zon file which is imported and comptime code builds the message structs and message validation functions from the imported zon (not sure if the latter is possible though).

The other way around (use Zig structs as the source of truth and derive the message format from those) may be tricky because Zig lacks custom attributes which would be needed to attach metadata to structs and struct items.

E.g. this would be a nice research project, but current Zig may lack features (it would be a nice design goal for the Zig language to enable such use cases without having to fall back to code generation).

2 Likes
3

should I add more information? what should be clarified?
or implement and then talk :grinning_face:

4

It doesn’t need them, we can reify type information at comptime in a format usable in runtime code. That’s basically how format functions work, if you squint.

To quote the IETF: I believe in rough consensus and running code. Show me some code.

4 Likes
5

Hmm, but how would that work inside a type definition, e.g. I’m thinking something like this (syntax is shitty though):

const Bla = struct {
    a: u32 @meta(.{ .ui_type = "color_picker", .color_space = "hsv" }),
    b: f32 @meta(.{ .ui_type = "slider", .min = 25.0, .max = 100.0, .step = 0.1 }),
} @meta(.{ .ui_type = "ui_attr_panel"});

…etc… the data inside @meta() would be .zon-style free form and attached to the type where it is accessible via comptime code (which then would need to be able to build a runtime function which describes the UI though).

…as I said, the other way around (use a .zon as source of truth and build types and functions from that) might make more sense.

…somehow I always end up at “Zig should be able to build functions with comptime code instead of just types” :thinking:

PS: building a UI with the description as comptime input param might work with the current features though…

PPS: I switched from ‘message protocol’ to ‘ui meta data’ because IMHO that’s a more common use case :wink:

1 Like
6

The way I do this kind of stuff is like so

The idea is to create a pub decl that contains meta options for the fields of the parent type. In the link above I define which fields should be skipped when serializing / deserializing (Ziggy is a data serialization format), but you could put there arbitrary options, like field name mapping, etc.

With this method you can do everything that hypothetical field attributes could do and this way you also force namespacing of these meta attributes, which is something that field attributes in, say, Go, usually don’t do, occasionally driving you into a corner.

6 Likes
7

the idea of zyamp is based on my already developed proprietary
application protocol , so i can say it’s very battle tested , but code

  • IP of the company
  • c++/c#

So regarding code - it will take some time :rofl:

Regarding comptime:

  • message consists of headers (similar to http) and body
  • one of the headers contains format of the body (json, proto, msgpack, raw etc.)
  • protocol does not care how the body/payload was created

I think comptime may be used for dealing with specific body representation, but it out of the protocol scope

8

Right on.

Looking forward to it.

I had in mind precisely:

So I will defer entirely to @kristoff’s post on that basis.

1 Like
9

I see, clever! And yeah, looks like this is actually more flexible than ‘meta-attributes’ - and you can have separate ‘attribute blocks’ for different ‘processors’. I need to chew on that idea for a bit :slight_smile:

1 Like