How to implement doorbell x86 without race conditions

I’m trying to make a mini scheduler where processes can send a message over a ringbuffer, then call ring() to wake up the receiving core if it was asleep, then that core services whatever it needs in it’s state.update() function. I’m using x86 umonitor and umwait as light sleep is what I’m going for. And I also added the if load on the ring, cause if I go with like fetchAdd generation u32, then it’d be a bunch of cacheline bouncing because ring() can be called a few million times/s from other cores as they submit messages to the main core. I’m pretty much completely lost in terms of properly implementing this with atomics. I think this is on the right track because I check update one more time before truly sleeping which is kinda Dekker’s algorithm, but I think its still wrong. Any help would be super appreciated, thanks.

const DoorbellState = enum(u8) {
    awake,
    asleep,
};

pub const Doorbell = struct {
    const Self = @This();

    state: mem.CachePadded(atomic.Value(DoorbellState)),

    pub fn init() Self {
        return .{ .state = .init(.awake) };
    }

    pub fn ring(self: *Self) void {
        if (self.state.load(.seq_cst) == .awake) {
            @branchHint(.likely);
            return;
        }

        self.state.store(.awake, .release);
    }

    pub fn prepare(self: *Self) void {
        self.state.store(.asleep, .seq_cst);
        hw.time.monitor(&self.state); // calls umonitor
    }

    pub fn cancel(self: *Self) void {
        self.state.store(.awake, .release);
    }

    pub fn commit(self: *Self, deadline: unit.Tick(u64, .n4)) void {
        if (self.state.load(.acquire) == .awake) return;
        hw.time.wait(deadline); // calls umwait
        self.state.store(.awake, .release);
    }
};
var last_time = hw.time.now().to(Tick(u32, .n4));
    while (true) {
        const now = hw.time.now(); // Tick(u64, .n4)
        const now_short = now.to(Tick(u32, .n4));
        last_time = now_short;

        const delay = state.update();

        if (delay.raw > min_sleep) {
            sys.doorbells[cpu].prepare();

            const delay2 = state.update();
            if (delay2.raw <= min_sleep) {
                sys.doorbells[cpu].cancel();
                continue;
            }

            const sleep_now = hw.time.now();
            sys.doorbells[cpu].commit(sleep_now.add(delay2.to(Tick(u64, .n4))));
        } else {
            core.atomic.spinLoop();
        }
    }

I don’t think there is enough context here to write a suitable answer. What are you using this for? What does state.update() do exactly?

From this, I am able to say however that the orderings (notably seq_cst) could probably be relaxed a bit here, and it should not be optimal to sleep immediately when there is no work. The CPU should only go to sleep if it truly has been idle for a while and can’t steal work.

I would suggest taking a look at the source code for std.Io.Queue, as it deals with the same sort of problems.

Internally, it uses a single Mutex and a Condition (1x per blocked reader/writer, which form a soudbly-linked list). They are both implemented using futexWait() and futexWake() and Atomics, which seems to be similar to what umwait does.

Oh, mb, this is a scheduler for a prototype kernel I’m making. I’m trying to keep it really simple, so for now I’m just sleeping as soon as there is no work cause the wakeup from umwait is pretty short and I’m not implementing work stealing either. state.update() does whatever the process is, so could be anything, but its purpose is check all inputs from other cores like ringbuffers then do work and decide if it still has work (return 0 wait) or can sleep for a while and returns that time.

How are you using umwait in a kernel? umwait is a usermode instruction

I’m doing most of my dev on linux for the time being with a cursed idea of a vm. I allocate 8GB mmap and fix it to a known virtual address (basically identity mapped) and then start 8 threads so they all see the 8gb pool and then pin them to my 8 cpu cores and my scheduler runs per core basically (I think later I could possibly get grub to boot with iso_cpus or something but idk). Rn I’m using io_uring to r/w raw ethernet packets and write directly to the sectors of the nvme for my filesystem, but I also have a working uefi target running in qemu with simple pcie drivers, but its more tedious to debug with. Its kinda like temple os where everything is ring 0, no kernel mode, which is why I can get away with running inside a linux process with no kernel privileges.

I would use signals to wake up cores (similar to IPIs on real hardware) then, and also look into real OS schedulers, because they typically don’t look like that

Yea it’s very different than the way most people do schedulers and kernels, but having cooperative non interrupt scheduling and a sideband kernel taking syscalls via ringbuffers is super fast and has a lot of advantages imo. I think IPIs are true interrupts so they push to the stack and run the interrupt routine so it’d choke the cpu.