Shimizu - The Wayland Protocol, in Zig

https://git.sr.ht/~geemili/shimizu

Hi all, I’ve been refining a library for working with Wayland protocols. I spun this project out of seizer, and I plan on switching seizer to depend on shimizu in the future.

There will probably be a few breaking changes to the API to make it more suitable for integration into an event loop (like libxev), improving the Listener type (it’s usable, but not great), or making it more server/client agnostic (it’s very much a client library at the moment).

Anyway, let me know if you want to see some specific examples or features.

18 Likes

I just split out the xkb library from seizer, and updated shimizu's readme to link to the new location: ~geemili/xkb - xkb_v1 keymap parsing and keycode translation, in zig - sourcehut git

Added another example and some screenshots

4 Likes

This seems like very cool project. I have 2 small suggestions: Instead of including wayland.xml in the source code, add wayland / wayland · GitLab into build.zig.zon (or just add url of wayland.xml once use case: ability to declare dependency on single file · Issue #17895 · ziglang/zig · GitHub is solved). Also, instead of including some of the other wayland protocols and generating
wayland-protocols module automatically, it would be nicer if you provided an example how to use generateProtocolZig function in build.zig so users could have control over which protocols they want.

1 Like

Thanks! I’ll definitely add an example of generating protocols and switch to downloading the protocols through the package manager. Not sure about removing the stable wayland-protocols. You need some protocols to do anything useful, and the examples do need xdg-shell. I’ll at very least make them a lazy dependency.

2 Likes

I’ve gone ahead and implemented your suggestions:

  1. shimizu now depends on wayland and wayland-protocols using the Zig package manager
  2. I’ve added a section on using shimizu-scanner to the README

Later I might add some options to replace the wayland.xml file and some documentation on doing that.

2 Likes

I’m new to vulkan and am trying to spin up vulkan the hard way without glfw.

I’m creating a request to vulkan to create a wayland surface, and I need *wl_display and *wl_surface objects.

Is there any way to get these objects via shimizu?

I did some digging and it looks like this struct is only defined in wayland-client.c, and appears to be something one can get via the protocol but isn’t defined by the protocol itself.

It looks like I can maybe get one with a .sync call?

I would love an example on how to get pointers to these structs with shimizu

wl_display is a specially defined object at id 1. You can acquire a proxy to it using shimizu.Connection.getDisplayProxy. However, this will not be the *wl_display Vulkan is expecting. Any references to Wayland objects in Vulkan will be using libwayland, and will require something that links libwayland, like zig-wayland to use.

It is possible to use shimizu and Vulkan together. I’ve done it for seizer, and you can see the crappy abstraction I made over it here. Note that this code has some cruft, and isn’t necessarily the most succinct. Here’s a high level overview of what needs to be done:

  1. Connect to the Wayland compositor.
  2. Acquire a Vulkan instance. This doesn’t require any knowledge of the Wayland compositor, you are just getting access to the GPU initially.
  3. Check if your Wayland compositor exposes a zwp_linux_dmabuf_v1 object from the linux-dmabuf-v1 protocol. We can make do without this extension, see the addendum for details.
  4. Check if your Vulkan device supports VK_EXT_external_memory_dma_buf. Same as above, we can make do without, see addendum for details.
  5. Create some images to render into, making sure they have dma_buf_bit_ext set. Normally you are instructed to use the Swapchain extension for this, however, we cannot, as it requires libwayland.
  6. Create wl_buffer objects using linux-dmabuf-v1 by exporting dma_buf handles from Vulkan.
  7. Tell Vulkan to render into one of images.
  8. Attach the associated wl_buffer to a wl_surface and commit it.
  9. Mark the Vulkan image/wl_buffer as in use until you receive a release event
  10. Grab another image/wl_buffer and repeat

And if that giant list didn’t scare you off, congratulations! :confetti_ball: You can now render to Wayland without linking libwayland! Unfortunately you will still need to link the appropriate libc.

Addendum: Rendering Without linux-dmabuf-v1

The linux-dmabuf-v1 protocol lets us tell the Wayland compositor about framebuffers/images that are located on the GPU. However, if that is unsupported, we can fall back to copying images from the GPU into main memory, and then passing that to the Wayland compositor. Here’s what to change:

4 Likes

I have begun this journey. I’m copy-pasting my way through Graphics.zig one piece at a time so that I can understand the journey that this is.

something something this is the most esoteric

1 Like

Good luck! Let me know if my code is too inscrutable and I’ll do my best to answer.

On another note, I’m going to try replacing libwayland in mach with shimizu. mach currently depends on the Vulkan swapchain extension, and after some discussion I decided that shimizu should support working with libwayland. So I added a separate output mode to the shimizu CLI and some functions to dlopen libwayland in a separate libwayland-compat branch.

Unfortunately shimizu’s API and the libwayland’s API are incompatible. I added an example using the libwayland API, and at some point I’ll update my mach pull request to use it.

I don’t love have the separation between the libwayland compatible and the pure Zig shimizu. If I could combine them that would be great, but at very least I would like to make both of them use the same generated code.

Anyway, once I’m done with the mach pull request I’ll merge the libwayland-compat changes into dev and make a tagged release.

1 Like

love mach, huge fan, only discovered shimizu because I seek specifically to break from all external dependencies because insanity.

This makes sense to do, but not requiring dlopen is a noble and insane path that I’d for sure like to see continued to be pursued

1 Like

Oh, for sure, I’m not planning on changing that anytime soon. shimizu will always have a way to use Zig in a statically linked binary.

Also unfortunately, graphics APIs require dynamically linking with the system libc. As I understand it, simply parsing and loading ELF files isn’t enough – at least glibc expects the process to have been started by it’s dynamic linker (/lib64/ld-linux-x86-64.so.2) or else it will cause a segmentation when a glibc function is called.

Right now I’m actually doing a rewrite of seizer to use software rendering for this reason. Its been interesting learning more about colorspaces and texture sampling, at very least.

I do wonder if the mesa would accept a proposal to make Vulkan work with statically linked executables. It might not be in scope for mesa, to be honest. It might be better to fix it on glibc’s end. Its frustrating that win32 might be a more stable ABI to target than Linux.

1 Like

This has always been a bit baffling to me. Could you link to any articles or material that explains how things ended up being this way?

1 Like

I don’t have any links at the moment (will try to find some) but here is what is in my head currently (I am probably wrong about some of this):

  • Windows, MacOS, and the BSDs all require linking to the system libc, as I understand it
  • Linux, as a project, only provides a kernel, and only specifies a syscall interface
  • The GNU project created glibc which provided a “system” libc from userland

Now is the part where this becomes a problem:

Coming back to the point about other systems that require libc, Windows and MacOS are both backed by large corporations that have the power to enforce using that libc, and also have the incentive and organizational capacity to maintain compatibility (although, less so with Apple). So when Vulkan says you should call dlopen("libvulkan.so.1"), this is not seen as an issue, and it solves several problems related to GPU drivers and multiple vendors. It’s only an issue on Linux because it’s the weird duck that doesn’t explicitly ship a system libc.

Add on top of that Linux is more popular as a server (where Windows and MacOS are mostly desktop) and you end up with some decisions being made that make sense for servers but less sense for desktops.

Some other links that are tangetially related:

Edit: Found some more interesting reads:

3 Likes

For Vulkan, unless you could somehow statically link user space drivers for the devices you want to support and select from them at runtime, you have to dynamically load them. Writing a loader to dynamically load glibc shared objects without dynamically linking against glibc seems to be an extremely difficult task.

In addition to the foreign-dlopen link provided above, the way that cosmopolitan libc does foreign dlopen from static binaries is pretty interesting (and a little horrifying imo). This article goes through making a dynamic loader that can run a dynamic ls executable (and shows why “manual” loading is so hard).

3 Likes


I’ve gotten my first vulkan window in my nixos hyprland environment up. Got everything to compile statically and run, but instead of loading vulkan directly as you do, I merged your code with my previous attempt at all this where i statically link vulkan-loader. Vulkan-loader finds the right driver on my nixos but can’t dynamically load it because my app is static compiled.

Annoyingly switched my app to target gnu instead of musl and its firing up, using a wayland window created by shimizu.

My hobby os can’t run this because of this dynamic loading glibc obstacle, shakes first at vulkan

2 Likes

Just curious, but is there any reason one might use this, over a wrapper around libwayland?

Reading through the thread, the main reasons one might use this package seem to be:

  • Closer to pure Zig. (if that’s a selling point for you)
  • Fully statically typed.
  • No dependency on libc. (related to both of the above)
  • Education.

But from the sound of things, this requires workarounds if you want to use it with libvulkan or other third-party libraries that expect libwayland (which might force you to depend on libc anyways…)

1 Like

Yep, you’ve got the gist of it.

Another reason to use shimizu (in the future, once I have libwayland-compat to a place where it’s stable) is cross compilation. As I understand it, libwayland doesn’t really support cross compilation at the moment.

I’m playing around with Vulkan for the millionth time, and decided to hand-roll a Wayland surface instead of using GLFW or similar. So far, shimizu seems fantastic – thanks very much!

@geemili since there’s no issue tracker for the project, I’d like to let you know about a bug I’ve just hit. Connection.close and Connection.recv use std.os.linux.recvmsg, with a TODO to switch to std.posix.recvmsg once it’s introduced. However, this has an awkward issue: if we link to libc, std.posix.errno expects that we actually used the (currently nonexistent) std.posix.recvmsg, so tries to read from libc errno instead, giving bogus results.

In my case, my Connection.close call is failing for some reason, but the return value is not -1, so std.posix.errno is returning .SUCCESS, and I instead get a safety panic on the @intCast(bytes_read)!

To fix this, either introduce your own recvmsg which copies the implementation from #19751 or something, or (for now) use std.os.linux.E.init instead of std.posix.errno.

(EDIT: having applied the latter patch locally, the error I was getting on Connection.close was just EAGAIN – so there wasn’t actually any problem in my code, this shimizu bug just made it seem like there was!)

I might end up using libwayland for now just so I can use VK_KHR_wayland_surface, but nonetheless, this library seems lovely.

2 Likes