Using the ziglang and setuptools-zig python extension to provide C extensions

If you are wondering where yesterday’s sudden change in daily downloads of the ziglang package on PyPI comes from, I am probably the one to blame.

This coincides with the upload, yesterday evening, for my package ruamel.yaml (with on average 3 million downloads a day over the last month) which switched from being dependent on pre-compiled wheels provided in ruamel.yaml.clib (with almost the same average number of downloads), to being dependent on ruamel.yaml.clibz.

ruamel.yaml.clibz depends, for building, on setuptools-zig and ziglang. And ziglang, and those two packages all see the same spike in downloads. I expect that today, the first full day of the changed dependency to be available, to push the ziglang downloads to over two million a day. There is of course daily fluctuation in downloads, and even thought today is Jan 1st, during Christmas 2025 there were more than two million daily downloads as well (of ruamel.yaml.clib).

This is currently (still) a C/Cython extension, no Zig code, but it is using the Zig toolchain for compilation and linking. With this change I can now start making enhancements to, and substitutions for, C code in Zig.

I am working on writing up a more extensive account of why I made this change (now) but it’s getting a bit too long for posting here, so I will probably post that under the yaml.dev domain, with a link here (a draft version of that account is here)

26 Likes

Holy cow, amazing work!

1 Like

Great stuff @Anthon. It’s called AllYourCodebase for a reason!

Looking forward to it. Pushing the boundaries of comptime to programmatically build the FFI layer between Zig code and pick-your-garbage-collected-language is a topic I’m keenly interested in, or anything reasonably adjacent to it.

Love to see it @Anthon. I think there’s a huge opportunity for Zig adoption via Python packaging.

Do you have a link to the repository? I’d like to see how you approached the build C with Zig and then use zig setuptools to create the Python package.

What were the biggest challenges you faced?

This is not using FFI, but you are probably already aware of that. Cython allows you to write a Pythonesque interface between your code and a C library. That gets Cython code gets compiled to C and provides the normal ABI interface that the Python executable expects for loadable extension modules.

1 Like

The repository is on sourceforge (sorry, I am a Mercurial user and don’t want to go back to git, although I might move to jujutsu): ruamel.yaml.clibz / Code / [a472ef]

The setup.py for that bunch of C files is below and should be useable as a template, at least for any stand-alone extension for Python written in C. The conditional setup_requires was necessary because otherwise these are downloaded/installed when just creating the source tarball as well.

There were no really big challenges, just a bunch of gruntwork, testing and adapting, and then having the guts to make the change and upload. I have had several Zig based Python extensions made for non-open source projects since 2020 that used setuptools-zig, and its readme has included examples for C, C+Zig and Zig only code for a long time. The ruamel.yaml.clib C code required adding a few parameters to zig build-lib:

zig build-lib -dynamic -fallow-shlib-undefined -femit-bin=/absolute/path/to/build/lib.linux-x86_64-cpython-314/_ruamel_yaml_ziglib.
cpython-314-x86_64-linux-gnu.so -lc -I /opt/util/tmp_yamlziglib/include -I /opt/python/3.14.0/include/python3.14 -O R
eleaseFast _ruamel_yaml_ziglib.c api.c writer.c dumper.c loader.c reader.c scanner.c parser.c emitter.c

and e.g. building on Linux alpine aarch64 (on my unifi firewall) I found out that installing ziglang gets you a zig file with 666 permission (if installed as build-requirement) wheras if you do a “normal” pip/uv install you get the execute permissions set (I easily wasted half a day on figuring that one out).

Testing is still a challenge as this is a vector with multiple dimensions: Python (6 versions); free-threaded (2, fortunately not for all Python versions supported (yet)); processor (2 that I tested, there are more); platforms (3 that I tested: Windows, macOS, Linux); architecture bit width (2: 32 and 64bit), libc/musl compatiblity (2). I did not test all the possible combinations, in that space but I covered each variant in each dimension at least once. And Linux & macOS (both 64bit) almost complete.

This is C compiling with zig, so Zig being at pre-1.0 was never a problem ( I had to make minor adjustments to the zig examples for setuptools-zig overtime of course, because of that).

import sys
from setuptools import Extension
from setuptools import setup

base = 'https://sourceforge.net/p/ruamel-yaml-clibz/'

setup(
    name='ruamel.yaml.clibz',
    version='0.3.4',
    python_requires='>=3.9',
    build_zig=True,
    author='Anthon van der Neut',
    author_email='a.van.der.neut@ruamel.eu',
    description='C version of reader, parser and emitter for ruamel.yaml, compiled with Zig,'
                ' derived from libyaml',
    long_description=open("README.md").read(),
    long_description_content_type="text/markdown",
    project_urls=dict(
        Home=base,
        Source=f'{base}code/ci/default/tree/',
        Tracker=f'{base}tickets/',
        Documentation='https://yaml.dev/doc/ruamel.yaml.clibz',
    ),
    license='MIT',
    ext_modules=[Extension(
        name='_ruamel_yaml_clibz',
        sources=[
            '_ruamel_yaml_clibz.c',
            'api.c',
            'writer.c',
            'dumper.c',
            'loader.c',
            'reader.c',
            'scanner.c',
            'parser.c',
            'emitter.c',
        ],
        extra_compile_args=[
            # '-O', 'Debug',
        ],
    )],
    setup_requires=[] if 'egg_info' in sys.argv else ['setuptools-zig>=0.5.1', 'ziglang<0.16'],
)
2 Likes

You’re right, I do know about the distinction between Cython and “cffi” extensions. The ABI interface is (as you are no doubt aware) the FFI, the Foreign Function Interface. But if one refers to Cython extensions as “FFI” people get confused, because colloquially there’s a distinction (a real one: a Cython project is not a cffi project or vice-versa). Language is messy :slight_smile:

I’m interested in seeing how much comptime can assist in taking Zig library code, and providing the expected export ABI for languages like Python, with as little drudgery and boilerplate as possible. In a perfect world, none.

I know this isn’t directly related to what you’ve done here, but I’m sure you can see why I’m interested nonetheless.

I added a link to the draft version of the more detailed account.

5 Likes

While it’s probably exciting to the community, with this move in ruamel.yaml you basically broke tons of CI jobs (and probably some local workstation workflows too)… ruamel.yaml is a dependency in GitHub - pre-commit/pre-commit-hooks: Some out-of-the-box hooks for pre-commit, and if previously it hasn’t required using the C compiler and python-dev libs installed on every machine and runner - now it is… So ppl urgently start to pin versions <0.19 in their setups ( New version on ruamel.yaml breaks the build system · Issue #1229 · pre-commit/pre-commit-hooks · GitHub ) or add tons of unneeded packages into their CI worker images (that’s the step we took to fix CI in hundreds of repos).

Today, a lot of ppl are still on holiday… Starting next week, I guess there will be a way more reports of broken CIs

1 Like

I am sorry to hear this broke CI jobs, I wasn’t aware that ruamel.yaml is used in github pre-commit hooks, and certainly not that this was done without using a pinned version of the package. I have advised to use pinning in the past, but I am not sure this was vocal and/or consistent enough.

During development I have tried to see if I can detect failure to install ruamel.yaml.clibz as a dependency for ruamel.yaml, and then fall back to ruamel.yaml.clib, but did not find a way to do this (which of course doesn’t mean there is no way).

I don’t know how Github is set up, but I do assume it has its own cache from where it pulls Python packages (instead of pulling from PyPI on each run). Would it not be easier to create the wheels for ruamel.yaml.clibz once and upload those to the cache (for each worker “type”), instead of extending all the workers.

3 Likes

Actually, it’s not a GitHub-specific feature… It’s a git feature… Basically, it allows running some pre-checks before creating a commit. Usually, it is used for automating linting (or to generate some files automatically). While the original intent was to use it on developer workstations, it has been adopted widely in CI jobs (not specifically github… it can be implemented with any CI engine, even Jenkins) to ensure that a project is properly linted.

As for the version pinning, while it can help to avoid the surprises like we had this New Year morning, it won’t actually solve the original problem - the requirement to build ruamel.yaml.clibz every time pre-commit run -a is run in a new CI runner (and nowadays, it is almost always an ephemeral runner, where every job starts in a fresh env).

Your CI automatically upgrades versions on your dependencies? Doesn’t that sound error prone or needlessly high risk?

Also, I saw your comment on gh, brigading isn’t cool dude.

4 Likes

Yep, sorry… I may be a bit harsh… It’s probably not the best day to see such surprises.

But, coming back to the original problem… The issue isn’t only in non-pinned dependencies (actually, in our internal jobs, we are pinning versions)

Even if pre-commit-hooks will use a pinned version, it doesn’t solve the problem… Early or later, it will need to bump to 0.19.0+ ver, and it still will face the original issue: since ruamel.yaml.clibz doesn’t have a binary distribution, every system (local or CI) will need to have build tools to use ruamel.yaml.

I’m sorry I don’t have a specific solution to your problem, but if you would like to go off on a tangent with an interesting related talk:

Some more information you may already know:

  • this problem is typically solved with some kind of caching in your CI runner, like installing packages once, saving to a persistent cache, and reusing the cache through multiple builds
  • the zig distribution on PyPi is designed for this use case (building binaries as needed and without system dependencies (zig bundles clang)). The zig 0.15.2 distribution is only approx. 50MB.
1 Like

For anyone finding this thread, and thinking they need to install gcc for ruamel.yaml==0.19.0 because of its dependency ruamel.yaml.clibz:

Please make sure you follow the recommendation in the README :

For this to function properly your Python (virtual) environment needs to have an up-to-date version of setuptools and wheels pre-installed.

See e.g. this issue that includes a Dockerfile and compose.yaml that make this work for python:3.13-slim-bullseye (which has no gcc installed)

4 Likes

Both wheels and setuptools seem to be required for building the package so defining them as dependencies should fix the issues people are having? Not sure what the best way for defining the build system dependency would be but maybe a pyproject.toml with the dependencies and defining setuptools as the build system

Unfortunately no. Before adding that line to the README, I tried adding setuptools as a dependency, but that won’t work. AFAICT setuptools is already loaded at the point the (build) dependencies are retrieved from the package, and I would actually have been surprised if any newer (or older) version would cause a re-import.

Welcome, two new people who are probably not here to talk about Zig.

That’s what we do on Ziggit. It’s not a place where we discuss broken build pre-commit hooks. sfyl and all but: showcase about using the Zig build system to make Python stuff happen, on topic. Triaging your issues with using the tool properly, off topic.

Feel free to stick around, but I’m getting out the mod stick and thwacking anyone who wants to talk about that here. It’s off topic.

2 Likes

Maybe you all should have used quicksilver for your high availability zig downloading needs!

1 Like

Well, that didn’t last.

1 Like