Mzg: A MessagePack library for Zig

Showcase
1

So I guess this is gonna be the 3rd MessagePack library here GitHub - uyha/mzg: A MessagePack library for Zig

5 Likes
2

Thanks for sharing! Welcome to the message pack club! :slight_smile:

Yeah I don’t think any of us have settled on a good way for customizations yet, and I didn’t even figure out how to correctly use the writer interface.

Also the writer interface is expected to change soon.

The way I did customizations was to generate a type for the value passed for serialization, which honestly was bad since it’s almost impossible for the user to figure out what options there are.

3

Yeah I don’t think any of us have settled on a good way for customizations yet

For customization, I just followed strictly the way that std.json does it. What surprised me the most was the adapters since I did not originally plan for it. They just kinda fell off the sky when I saw that I can simply check for mzgPack/mzgUnpack at the very beginning. I was originally planning to wrap containers like std.ArrayList in a struct like

fn ArrayAdapter(comptime T: type) type {
  return struct {
    container: std.ArrayList(T),
    // More details
  };
};

but then the users have to access the data like adapter.container.items which I did not like at all, and with the current approach, they wrap the containers themselves and the containers get modified directly, which I like very much.

and I didn’t even figure out how to correctly use the writer interface.

I’m not sure how to use it correctly also, so I just settled on using only the writeAll function, so that if something changes, I can change it easily.

4

After dogfooding this library, I added some more functionality.

  1. packing and unpack now functions that accept a map of adapters so callers do not need to explicitly create the adapters themselves. Example:
    const pack_map = .{
    .{ std.ArrayListUnmanaged([]const u8), mzg.adapter.packArray },
    .{ std.ArrayListUnmanaged(u8), mzg.adapter.packArray },
    .{ std.StringArrayHashMapUnmanaged([]const u8), mzg.adapter.packMap },
    .{ std.StaticStringMap([]const u8), mzg.adapter.packMap },
};
pub fn pack(
    value: anytype,
    writer: anytype,
) mzg.PackError(@TypeOf(writer))!void {
    return mzg.packAdapted(value, pack_map, writer);
}

const unpack_map = .{
    .{ std.ArrayListUnmanaged([]const u8), mzg.adapter.unpackArray },
    .{ std.ArrayListUnmanaged(u8), mzg.adapter.unpackArray },
    .{ std.StringArrayHashMapUnmanaged([]const u8), mzg.adapter.unpackMap },
    .{ std.StaticStringMap([]const u8), mzg.adapter.unpackStaticStringMap },
};
pub fn unpack(
    buffer: []const u8,
    out: anytype,
) mzg.UnpackError!usize {
    return mzg.unpackAdapted(unpack_map, buffer, out);
}
  2. unpacking now has functions that accept an allocator so that memory allocation can happen during unpacking. Example:
    const unpack_map = .{
    .{ std.ArrayListUnmanaged([]const u8), mzg.adapter.unpackArray },
    .{ std.ArrayListUnmanaged(u8), mzg.adapter.unpackArray },
    .{ std.StringArrayHashMapUnmanaged([]const u8), mzg.adapter.unpackMap },
    .{ std.StaticStringMap([]const u8), mzg.adapter.unpackStaticStringMap },
};
pub fn unpackAllocate(
    allocator: Allocator,
    buffer: []const u8,
    out: anytype,
) mzg.UnpackAllocateError!usize {
    return mzg.unpackAdaptedAllocate(allocator, unpack_map, buffer, out);
}
1 Like
5

no offence

I checked several zig native (not c lib based) implementations:

None of these implementations check for compatibility with another programming language

There is Command-line tools for converting between MessagePack and JSON

My be try to use this or similar tool for the test?

6

I’m not sure what you mean by “compatibility with another programming language”. MessagePack is a binary format, libraries that encode/decode it do just that, convert objects of a programming language to/from MessagePack according to the spec. MessagePack itself doesn’t have a test suite to check for compliance also, so all libraries just do it on the best effort basis.

7

As for converting between json and msgpack, I already have a branch to try to do that, it’s just a bit tedious to get all the data structure right.

8

I meant use case when msgpack blob transfered between
zig client and go server or similar …

9

I added a small example for packing in Zig and unpacking in Python. I’m still not sure if that’s what you are asking for that or you are asking why should one use MessagePack instead of JSON. If it’s the latter, you can refer to https://msgpack.org/ to see the reasoning.

2 Likes
10

yes, i asked about exchange between zig and another programming language

11

I also added an example to take MessagePack bytes from Python and deserialize them in Zig. Hope it helps.

2 Likes