Preventing page faults due to stack expansion

Help
1

I’m implementing some realtime software and I want to avoid page faults. One way is to lock my programs memory using the mlockall syscall.

I can lock my currently allocated memory and all future allocations, which is great (new allocations could fault once and then never fault again).

But I have designed my program to only allocate on startup, lock memory, then begin operation. Theoretically I should never page fault.

However, there is a potential issue with the stack. If my stack grows into a paged region, then a page fault will occur.

Reading man mlockall, they recommend a solution:

Real-time processes that are using mlockall() to prevent delays on
page faults should reserve enough locked stack pages before
entering the time-critical section, so that no page fault can be
caused by function calls. This can be achieved by calling a
function that allocates a sufficiently large automatic variable
(an array) and writes to the memory occupied by this array in
order to touch these stack pages. This way, enough pages will be
mapped for the stack and can be locked into RAM. The dummy writes
ensure that not even copy-on-write page faults can occur in the
critical section.

What would this look like in zig? And how can I verify that it works?

If Builtin function to tell you the maximum stack size of a given function · Issue #157 · ziglang/zig · GitHub is implemented, I could potentially touch every byte of my stack to prevent page faults :slight_smile:

2

In release safe, you can use something like this:

While zig doesn’t give you actual static size of a function, there’s stack size parameter for an exe at the build system level, and you can use that.

3

Isn’t it not enough to probe only a single byte? Shouldn’t you probe all the bytes? If you only probe a single byte might you skip over a paged page?

The man page I linked says:

The dummy writes
ensure that not even copy-on-write page faults can occur in the
critical section.

I interpret this to mean that I need to write to all the pages.

I came up with this implementation:

pub noinline fn probeStack(comptime size: usize) void {
    var big: [size]u8 = undefined;
    const big_ptr: *volatile [size]u8 = &big;
    big_ptr.* = @splat(0xbb);
}

test "probe stack" {
    probeStack(2_000_000);
}

// crashes, stack overflow
// test "probe stack, overlow" {
//     probeStack(100_000_000);
// }

pub fn exampleUsage() !void {

    // all current pages are faulted
    // all future pages only fault once
    try mlockall(.{ .CURRENT = true, .FUTURE = true });
    
    // potential page faults here
    // or SIGSEGV here due to out of memory, exceeding memory limits of program
    probeStack();  

    // no more page faults from here on
    doCriticalOperations();
}

Details:

  • noinline: prevent inlining so that the stack retracts after probing. We dont want to probe useless inlined stack memory and then the remaining program uses memory outside of our probe.
  • volatile: prevent optimizing compiler from thinking these memory writes don’t have side effects and deleting them.
  • splat: write to all the bytes, so that all potential stack pages fault into RAM.
4

Is there a way to do this with runtime known size?

5

Not without inline assembly since alloca was removed.