Using the ziglang and setuptools-zig python extension to provide C extensions

If you are wondering where yesterday’s sudden change in daily downloads of the ziglang package on PyPI comes from, I am probably the one to blame.

This coincides with the upload, yesterday evening, for my package ruamel.yaml (with on average 3 million downloads a day over the last month) which switched from being dependent on pre-compiled wheels provided in ruamel.yaml.clib (with almost the same average number of downloads), to being dependent on ruamel.yaml.clibz.

ruamel.yaml.clibz depends, for building, on setuptools-zig and ziglang. And ziglang, and those two packages all see the same spike in downloads. I expect that today, the first full day of the changed dependency to be available, to push the ziglang downloads to over two million a day. There is of course daily fluctuation in downloads, and even thought today is Jan 1st, during Christmas 2025 there were more than two million daily downloads as well (of ruamel.yaml.clib).

This is currently (still) a C/Cython extension, no Zig code, but it is using the Zig toolchain for compilation and linking. With this change I can now start making enhancements to, and substitutions for, C code in Zig.

I am working on writing up a more extensive account of why I made this change (now) but it’s getting a bit too long for posting here, so I will probably post that under the yaml.dev domain, with a link here (a draft version of that account is here)

15 Likes

Holy cow, amazing work!

1 Like

Great stuff @Anthon. It’s called AllYourCodebase for a reason!

Looking forward to it. Pushing the boundaries of comptime to programmatically build the FFI layer between Zig code and pick-your-garbage-collected-language is a topic I’m keenly interested in, or anything reasonably adjacent to it.

Love to see it @Anthon. I think there’s a huge opportunity for Zig adoption via Python packaging.

Do you have a link to the repository? I’d like to see how you approached the build C with Zig and then use zig setuptools to create the Python package.

What were the biggest challenges you faced?

This is not using FFI, but you are probably already aware of that. Cython allows you to write a Pythonesque interface between your code and a C library. That gets Cython code gets compiled to C and provides the normal ABI interface that the Python executable expects for loadable extension modules.

1 Like

The repository is on sourceforge (sorry, I am a Mercurial user and don’t want to go back to git, although I might move to jujutsu): ruamel.yaml.clibz / Code / [a472ef]

The setup.py for that bunch of C files is below and should be useable as a template, at least for any stand-alone extension for Python written in C. The conditional setup_requires was necessary because otherwise these are downloaded/installed when just creating the source tarball as well.

There were no really big challenges, just a bunch of gruntwork, testing and adapting, and then having the guts to make the change and upload. I have had several Zig based Python extensions made for non-open source projects since 2020 that used setuptools-zig, and its readme has included examples for C, C+Zig and Zig only code for a long time. The ruamel.yaml.clib C code required adding a few parameters to zig build-lib:

zig build-lib -dynamic -fallow-shlib-undefined -femit-bin=/absolute/path/to/build/lib.linux-x86_64-cpython-314/_ruamel_yaml_ziglib.
cpython-314-x86_64-linux-gnu.so -lc -I /opt/util/tmp_yamlziglib/include -I /opt/python/3.14.0/include/python3.14 -O R
eleaseFast _ruamel_yaml_ziglib.c api.c writer.c dumper.c loader.c reader.c scanner.c parser.c emitter.c

and e.g. building on Linux alpine aarch64 (on my unifi firewall) I found out that installing ziglang gets you a zig file with 666 permission (if installed as build-requirement) wheras if you do a “normal” pip/uv install you get the execute permissions set (I easily wasted half a day on figuring that one out).

Testing is still a challenge as this is a vector with multiple dimensions: Python (6 versions); free-threaded (2, fortunately not for all Python versions supported (yet)); processor (2 that I tested, there are more); platforms (3 that I tested: Windows, macOS, Linux); architecture bit width (2: 32 and 64bit), libc/musl compatiblity (2). I did not test all the possible combinations, in that space but I covered each variant in each dimension at least once. And Linux & macOS (both 64bit) almost complete.

This is C compiling with zig, so Zig being at pre-1.0 was never a problem ( I had to make minor adjustments to the zig examples for setuptools-zig overtime of course, because of that).

import sys
from setuptools import Extension
from setuptools import setup

base = 'https://sourceforge.net/p/ruamel-yaml-clibz/'

setup(
    name='ruamel.yaml.clibz',
    version='0.3.4',
    python_requires='>=3.9',
    build_zig=True,
    author='Anthon van der Neut',
    author_email='a.van.der.neut@ruamel.eu',
    description='C version of reader, parser and emitter for ruamel.yaml, compiled with Zig,'
                ' derived from libyaml',
    long_description=open("README.md").read(),
    long_description_content_type="text/markdown",
    project_urls=dict(
        Home=base,
        Source=f'{base}code/ci/default/tree/',
        Tracker=f'{base}tickets/',
        Documentation='https://yaml.dev/doc/ruamel.yaml.clibz',
    ),
    license='MIT',
    ext_modules=[Extension(
        name='_ruamel_yaml_clibz',
        sources=[
            '_ruamel_yaml_clibz.c',
            'api.c',
            'writer.c',
            'dumper.c',
            'loader.c',
            'reader.c',
            'scanner.c',
            'parser.c',
            'emitter.c',
        ],
        extra_compile_args=[
            # '-O', 'Debug',
        ],
    )],
    setup_requires=[] if 'egg_info' in sys.argv else ['setuptools-zig>=0.5.1', 'ziglang<0.16'],
)
1 Like

You’re right, I do know about the distinction between Cython and “cffi” extensions. The ABI interface is (as you are no doubt aware) the FFI, the Foreign Function Interface. But if one refers to Cython extensions as “FFI” people get confused, because colloquially there’s a distinction (a real one: a Cython project is not a cffi project or vice-versa). Language is messy :slight_smile:

I’m interested in seeing how much comptime can assist in taking Zig library code, and providing the expected export ABI for languages like Python, with as little drudgery and boilerplate as possible. In a perfect world, none.

I know this isn’t directly related to what you’ve done here, but I’m sure you can see why I’m interested nonetheless.

I added a link to the draft version of the more detailed account.

2 Likes