Wrapping C callbacks where the C callback has no extra "opaque/context" void* parameter

Help
,
1

Hello,

Objective: To build a Zig wrapper around a C-lib. This C-lib has callbacks that can be registered and fired from the C-based code. However, I’d like end-users of the Zig wrapper to not have to concern themselves with the callconv(.C) callbacks and just write Zig callbacks. Internally in my Zig code, it would somehow register true C-ABI callbacks and then call the end-user’s “nicer” Zig callbacks where in some cases the params are different, or cleaned up.

There was an older question I stumbled upon that has a great answer on how to wire up a C callback, which then later calls some Zig code. In the original post, the user needed to call their Zig function which was declared on a method so they needed access to the self parameter.

Here’s the original post: How to wrap a Zig function to be called from C?

My question is related: I’d like to accomplish the same thing but in my case the C callback in a third-party lib, doesn’t have an opaque void* extra parameter that I can utilize to pass additional context.

Without going down the rabbit hole of global variables or something in static memory am I totally out of luck here? (The c-lib does do some internal multi-threading so global or static memory is likely a bad idea anyway)

I can’t seem to come up with any “closure” based solution because the C callback ultimately has no way of referencing additional context.

For reference here is what the C lib callback looks like and its purpose is for logging. It will be called from C code when I register a user-provided callback. But I want the user of this library to not have to write any callconv(.C) callbacks and write Zig callbacks so I’m trying to get the C callback to dispatch to the Zig callback. I am trying to provide a Zig lib that wraps the C in a more idiomatic style of code.

const ConfLogCallbackCABI = *const fn (c_int, [*c]const u8, [*c]const u8) callconv(.C) void;

First arg is just a integer log level, 2nd is a string which would be like a log class, third is the log message.

I’m aware of Zig’s limitations of not having first class support of true closures because Zig must not have any hidden allocations. I just don’t know if I can do what I want to do or if the user must define their callbacks in the C calling convention if there is no way around this.

2

Just brainstorming around here and I am not sure if it is a idea that would work well…, but here is it anyway.

One option might be to just pre generate some number of functions (maybe using comptime) and have them call into a handleCallback(some_hardcoded_index, other_args ...) function the index then can be used to store and retrieve the associated data, so that would basically give you an arbitrary but predetermined amount of max possible callbacks possible concurrently. Basically giving out every callback function pointer only once and then only returning that one to the pool when it was called / there aren’t any further callbacks expected.

I guess what I describe can be described as a global variables solution, because you basically would have an array of functions that only differ in their internal index and then some array of context slots.

Maybe you also could use dynamic libraries, or mmap and dynamic code generation to do similar shenanigans at runtime, but it would still be based on placing things in memory, seems unavoidable to me, but maybe there is some trick I don’t know.

1 Like
3

Thank you @Sze, something along these lines did vaguely cross my mind but I was trying to avoid this ultimately. As you pointed out there may be some implications here with global state even if it’s somewhat a bucketed or shaded solution.

I was hoping there was Zig magic that I can exploit…I was just noodling around with @fieldParentPtr but that won’t work for me either because it needs to reference a field that belongs to the struct in question but since the C callback has no possible of way of referencing anything outside of the pass parameters it might be that I have no solution that avoids global or static state. Yecch.

If anyone else has thoughts, I’m all ears but I recognize this might be just a truly fundamental design flaw with this particular C-lib that I am wrapping. For the record, some of the callbacks in this library do have the extra opaque parameter allowing me to set anything I need. Interesting choice, that some callbacks have it and others don’t.

1 Like
4

A program = data + algorithms. The function is the algorithm, but the data must be supplied by either its arguments or global data. If the arguments are not sufficient, then there’s only global data. No language can compensate for this, not even those with closures.
With that said, you can always retrieve custom data from the arguments, as long as the library gives you back something that you gave it ealier.

I’m assuming that the last argument is a null terminated string that you provided and the library will print it, but not modify it. In that case, just compute where your data is based on this pointer, like this:

const Data = struct{
  msg: [2:0] = .{'h', 'i'},
  payload: u8 = 2,
};

fn callback(_ : c_int, [_: *c]const u8, str: [*c]const u8) callconv(.C) void{
  const data: *const Data = @fieldParentPtr(str, "msg");
  // You can now access your payload.
}

pub fn main() void{
  const data: Data = .{};
  callLibrary(callback, &data.msg);
}

If neither of the pointers that are passed to the function are provided by you, then the only solution is global data.

5

@LucasSantos91 :No language can compensate for this, not even those with closures.

Except that, that is literally one of the things closures can do. See this partial Go-based example:

func main() {
	var f = new(Foo)
	var c = func(a, b int) {
        // This callback captures f which is a pointer to a struct with a doSomething method.
		f.doSomething()
	}
}

Anyways, I was just trying to see what possibilities I would have with Zig code.

6

In cases like Go, the language is simply hiding the context behind syntax sugar and hidden data and control flow. The context data is still there under the hood. In your case, you have a C lib, which of course has no understanding of closures – just C function pointers and arguments, where there is no place to hide context (unless you can smuggle via parameters, perhaps as an index or lookup key, as others have noted)

4 Likes
7

Yes, it’s become more clear to me at least for Zig or C that this is the case. I was just pointing out that other languages with closures do allow you to indeed capture additional things as needed.

This isn’t always a good idea but it does work. I just need to start thinking at a lower level going forward.

8

If you interfaced with a higher level language via some other ABI then that could theoretically have a mechanism to automatically manage additional context that allows you to implement something closure like.

So in a way I agree with you, but as long as you are stuck with interacting with things via CABI then you are stuck with more manual things.

In this case I am leaning towards giving the c library the fault for not giving you an easy way to carry the needed additional context.

Personally for me that would be a reason to put that library on the list of dependencies I try to eliminate or replace in the future, or if it is too much work you could try and fork/patch it.

1 Like
9

Thanks @Sze.

I realized today that this particular C lib is huge and quite complicated and does have some long-standing tech-debt.

They maintain a strong backwards compatibility promise to not break the api but this particular logging method was a missed opportunity to supply one extra opaque parameter but I realized earlier today they do provide a workaround. All of their newer callbacks do indeed have this opaque parameter so they realized it was beneficial and needed in some cases.

The workaround is ugly but it does work for me because there are additional methods to set/get an opaque object off of the root C object that runs the show.

Anyways, if anything this was all a good exercise in greater understanding of things.

I really like Zig’s no-stance on hidden memory allocation and I get the need to draw a hard line here.

1 Like