Is passing `std.mem.Allocator` as `?*const anyopaque` to C considered safe?

Help
1

Hello,

I’m trying to create a simple C API similar to Lua VM’s API, where anyone can embed the language from any C application.

I wanted to use Zig’s allocators within C, such that the memory useage remains deterministic, and can be tested (with std.heap.GeneralPurposeAllocator’s memory leak detectors.

Since we can’t store std.mem.Allocator in an extern struct, I converted it to an ?*const anyopaque pointer and stored it within the struct.

The code is shown below and seems to work. It can detect memory leaks, double frees, etc.

Though the question is,

  • Is this safe to do?
  • (or if this is unsafe, Is there any resources I can follow to do this safely? )

In the end I would want to be able to pass the ExperimentalAllocator to C API functions, such that those functions could manage memory similar to Zig.

I know it seems potentially risky, but it needs to be done.

Thanks.

/// Allocator that can be used in C.
///
/// To use this allocator,
/// - Create the allocator in C.
/// - Pass the allocator to the required functions.
///
/// Note: For memory safety, each individual instance of the runtime,
/// and each thread invoked by each instance of the runtime,
/// will require a separate allocator.
const ExperimentalAllocator = extern struct {
    context: ?*const anyopaque,
};

pub export fn ExperimentalAllocatorInit() ExperimentalAllocator {
    var gpa = std.heap.GeneralPurposeAllocator(.{ .safety = true }){};
    const allocator = gpa.allocator();
    return .{ .context = &allocator };
}

pub fn ExperimentalAllocatorFree(allocator: *const ExperimentalAllocator, slice: anytype, num_bytes: usize) void {
    const a: *const std.mem.Allocator = @ptrCast(@alignCast(allocator.context.?));
    const s = @as([*]u8, @ptrCast(slice))[0..num_bytes];
    a.free(s);
}

fn dev(allocator: std.mem.Allocator) !void {
    const a = ExperimentalAllocator{ .context = &allocator }; // Is this safe?
    const slice = try allocator.alloc(u8, 1000);
    ExperimentalAllocatorFree(&a, slice, 1000); // < Is this safe?
}
1 Like
2

I really don’t trust the code, where I take references, such as,

  • &allocator where allocator is const allocator = gpa.allocator();

This allocator seems like a stack allocated object, whose lifetime exists, as long as the function exists.

It seems unsafe if I take its reference, because since it’s on stack, it could be overwritten by something else…

3

You are correct about your code being unsafe because it references stack memory.

Use global variables, it’s the easiest and least sketchy solution.

1 Like
4

Thank you for your advice.

Making it global will solve the issue for now, but will cause more issues in future, as you might already foresee. Especially when we would want to create multiple allocators, in a single application.

What if we did something more sketchy?

The issue is that we can not store a non extern struct inside another extern struct.

So, what if we instead do some bit hacking to copy the bytes of the std.mem.Allocator struct, and store it as a []u8 array?

In C that would be similar to storing an array of bytes as char []. And since it’s just a bunch of bytes, we can store anything in it. Including non extern structs.

This would ensure that we’re only storing the data of std.mem.Allocator and we won’t have to store a bug prone pointer to stack memory.

In pseudocode,

const MAX_ALLOCATOR_SIZE = 512;

const ExperimentalAllocator = extern struct {
  context: [MAX_ALLOCATOR_SIZE]u8
};

// then ...

const allocator = gpa.allocator();
return ExperimentalAllocator {.context = CopyBytes(allocator)}
5

I did think about that, its sketchy but there isnt any other way to give the C code ownership of the memory,

Btw, you will also have to do this with the allocator implementations (DebugAllocator, etc) if you go this route.

1 Like
6

I did it.

We can now allocate slice and free it using our C compatible allocator.

/// Allocator that can be used in C.
///
/// To use this allocator,
/// - Create the allocator in C.
/// - Pass the allocator to the required functions.
///
/// Note: For memory safety, each individual instance of the runtime,
/// and each thread invoked by each instance of the runtime,
/// will require a separate allocator.
const NxnAllocator = extern struct {
    context: [@sizeOf(std.mem.Allocator)]u8,

    const Self = @This();

    fn alloc(self: Self, comptime T: type, n: usize) ![]T {
        const allocator = std.mem.bytesAsValue(std.mem.Allocator, &self.context);
        return try allocator.alloc(T, n);
    }

    fn free(self: Self, memory: anytype) void {
        const allocator = std.mem.bytesAsValue(std.mem.Allocator, &self.context);
        allocator.free(memory);
    }
};

fn dev(allocator: std.mem.Allocator) !void {
    const a = x: {
        var a = NxnAllocator{ .context = undefined };
        const bytes = std.mem.asBytes(&allocator);
        for (bytes, 0..) |value, i| {
            a.context[i] = value;
        }
        break :x a;
    };

    const slice = try a.alloc(u8, 1000);
    defer a.free(slice);
}

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{ .safety = true }){};
    const allocator = gpa.allocator();

    try dev(allocator);

    const leaked = gpa.detectLeaks();
    if (leaked) {
        std.debug.print("Has memory leaks: {any}\n", .{leaked});
    }
}
7

you should align the field to the alignment of std.mem.Allocator

fyi you can

const a = NxnAllocator{
 .context = std.mem.asBytes(&allocator).*,
};

asBytes returns a pointer to an array of the size of the type.

idk what NxnAllocator means, I’d call it ExternAllocator.

again, you’d need to do the same trick with the implementation types.

1 Like
8

This is a really helpful advice. I will do it.

Edit: I suppose this should be enough?

const ExternAllocator = extern struct {
    context: [@sizeOf(std.mem.Allocator)]u8 align(@alignOf(std.mem.Allocator)),

    const Self = @This();

    fn init(allocator: std.mem.Allocator) Self {
        var self = ExternAllocator{ .context = undefined };
        const bytes = std.mem.asBytes(&allocator);
        for (bytes, 0..) |value, i| {
            self.context[i] = value;
        }
        return self;
    }

    fn alloc(self: Self, comptime T: type, n: usize) ![]T {
        const allocator = std.mem.bytesAsValue(std.mem.Allocator, &self.context);
        return try allocator.alloc(T, n);
    }

    fn free(self: Self, memory: anytype) void {
        const allocator = std.mem.bytesAsValue(std.mem.Allocator, &self.context);
        allocator.free(memory);
    }
};
1 Like