Msgpack library

Showcase
,
1

As part of a project, on which I’m learning Zig, I implemented a library for working with msgpack. There is one alternative msgpack library for Zig, but that did not fit my needs, as it tries to be dynamic. My library is very static and low level. I need it to be fast and encode low level data structures exactly as I want without overhead of constructing variant objects.

It’s not a standalone project for now, and there are no guarantees on the API yet, but if there is interest, I can turn it into a library that can be easily included into build.zig

Comments on the API are very welcome.

10 Likes
2

Can’t wait to give this a try in a few days, if you would like help packaging it for the zig package manager I am happy to help.

It’s fairly common for me to use static msgpack formats and redis for effectively interprocess communication / queues.

3

It’s great to see another Zig developer working on a static, performance-focused msgpack implementation.

The focus on static, low-level implementation that handles data structures without unnecessary overhead sounds exactly like the approach I’m taking, although my project is primarily for learning purposes and I don’t plan to maintain it long-term.

I look forward to checking out your code once my implementation is more mature. I’m curious to see the different approaches we took to solving a similar problem. Feel free to share more details about your library as it progresses!

4

I’ve updated the main API to match the conventions from std.json and moved the code to a separate repository. API is still not stable, I’m open to comments, as I’m still learning what’s typical in Zig. Code works on 0.13, not the development version of Zig.

2 Likes
5

Love msgpack format, great to see this project; been using it for a few years with msgspec and litestar in Python land.

6

Released version 0.5.0 with zig 0.15 support and msgspec-compatible encoding of tagged unions.

3 Likes
7

Nice! Looking forward to checking this out.

8

Release version 0.6.0 with support for the new std.Io.Reader/Writer interfaces. The main benefit is smaller binary size, as we it no longer needs to deal with anytype and also for null handling, it’s beneficial that the code can peekByte() on any reader.

1 Like
9

Great library, have you tried benchmarking it?

I’d like to know a performance comparison between these two.

10

Oh, your library is static, great, the library zig-msgpack of zigcc is dynamic parsing

I believe dynamic performance is inferior to static performance.

But I still want to know how much the performance will be worse.

1 Like
11

I’ve not benchmarked my library, but I expect it to be faster than any dynamic solution.

I use it in a very performance sensitive project and it’s fine.

But I’d welcome an external benchmark.

12

Thanks to a new contributor, banteg, we now have a new release:

Added

  • Support for Zig 0.16

Fixed

  • Binary data is now encoded with the correct bin8/bin16/bin32 msgpack headers instead of string headers; string headers are still accepted when decoding for backwards compatibility
  • Array and map header size calculation incorrectly included a non-existent u8 size tier; arrays/maps with 16-65535 elements now correctly use the array16/map16 3-byte header
  • sizeOfPackedAny now correctly handles optional values and propagates errors from string/array size calculations
  • Custom formats (msgpackFormat, msgpackFieldKey, msgpackRead, msgpackWrite) now work correctly when the type is wrapped in an optional
1 Like