Setting up maintainable CI

Help
1

I’m setting up a self-hosted CI runner for a zig project targeting Linux. One wrench to throw in is that it needs a special kernel installed (PREEMPT_RT), and some OCI container runtime (docker/podman etc) (because I also target that environment).

Anyone have any tips on how to set this up so that each time I come back to the project every few months it’s not just broken and I can actually maintain it?

Some annoyances that I have had in the past:

  1. Upgrading packages breaking my networking configuration.
  2. Upgrading packages breaking docker.
  3. Not having the latest kernel / having to use some sketchy kernel build from a 3rd party (because it’s hard to build the kernel)

Ideally I could define the full config of my device in my repo, from kernel to bios settings etc.

4 Likes
2

With docker you don’t need to recreate your environment, so I think you would be helped if do not start from scratch every time. I used to have a Dockerfile that created the environment, that was then re-used by mapping in the source code from the host and doing the testing. For me, as I was not testing against low level operating features, there was seldom need to save the image (or recreate it once I had all the necessary tools installed).

I am not sure what you are currently using, but it might be that you have only a compose.yaml file that pulls in a fresh image indescriminately. That is not how I used to do things. I have a Dockerfile that creates the image that I need (which sometimes needs a bit of trial and error to create), then a composel.yaml file that maps in the sources that I actually want to test (using volumes).

If I don’t explicitly force a rebuild of the image ( docker compose build --pull) then nothing can break things apart from changes in my software. In my experience docker itself (on the host) has always been backwards compatible.

If you need to do a partial upgrade of the image, make sure you don’t start your Dockerfile with some FROM ...:latest, but use a specific version. You can also install packages in a two step process (with those packages that never change in the first step). Docker will then only rebuild overlays starting from the lines have changed (faster). I fully rely on the overlays images being cached by docker itself, and only saved/export these when moving to a new server (and experiencing trouble rebuilding because some repo for source or packaging was no longer available).

My compose.yaml files all look something like

services:
  myapp:
    container_name: ruamel_yaml_clib   # <= your unique container name
    build: .     # i.e. build the Dockerfile in the current directory
    volumes:
      - ..:/src:rw    # map in the current directory as /src in the image for read and write (so you can get results out if needed)
1 Like
3

I’d pin an official PREEMPT_RT line and sit on it.I dunno if you need to chase the latest greatest every month if RT matters. You could just pick a good RT branch from the official kernel.org RT patch trees and jsut pin that exact kernel/patch combo in the repo. kernel.org is still publishing RT patch series for maintained branches such as 6.6 and 5.10, which is obvy alot better than depending on some rando third-party kernel build.

2 Likes