Question about declaration-level metaprogramming: generating aliases from reflected decls

Help
,
1

Hi all,

I ran into a limitation while working on a lua.zig wrapper, and I want to check whether I’m understanding Zig’s design correctly, or whether I’m missing an intended approach.

My practical use case

I have a namespace like this:

const APIs = struct {
    pub extern fn lua_createtable(L: *State, narr: c_int, nrec: c_int) callconv(.c) void;
    pub extern fn lua_pushinteger(L: *State, n: c_int) callconv(.c) void;
    // ...
};

What I want for the public API is:

pub const createtable = APIs.lua_createtable;
pub const pushinteger = APIs.lua_pushinteger;
// ...

So that users (remain parts of my code) can simply write:

const lua = @import("lua.zig");
lua.createtable(...);
lua.pushinteger(...);

(I know there is zlua project in github, but it required to build lua itself into shared library, which is not fill my purpose)

This is currently working fine if I write those aliases manually.

What I hoped to do

Since all the source declarations are already present in APIs, I hoped I could:

  1. iterate over @typeInfo(APIs).@"struct".decls
  2. get each declaration name and member
  3. strip the lua_ prefix (or transform luaL_... as needed)
  4. somehow generate: pub const {new_name} = {member};

But as far as I can tell, Zig currently allows reflection over decls, and allows type construction for fields via @Struct, but does not allow constructing new declarations or injecting them into a container/module scope.

So the only current options seem to be:

  • write aliases manually
  • or generate a .zig file with codegen modified it (I have other helpers in lua.zig, so I could just replace parts of file, not whole).

My questions

1. Is this limitation intentional because of Zig’s “No hidden control flow” / anti-macro philosophy?

I’m wondering whether declaration generation is intentionally excluded because introducing new declarations from comptime would be considered too close to hidden language-level metaprogramming.

In other words: is the inability to generate decls from reflected decls a deliberate design boundary, rather than just an unimplemented feature?

2. Is there any roadmap or long-term plan for declaration-level metaprogramming?

I’m not necessarily asking for a full macro system, but something like:

  • declaration reification
  • constructing container decls in @Struct
  • or some supported way to generate alias declarations from reflected declarations
    Has this been discussed as a possible future direction, or is it considered out of scope?

3. In cases like this, is code generation the recommended Zig-style solution?

For this kind of FFI wrapper ergonomics problem, is the intended answer simply:

“yes, use build-time codegen and generate a .zig file”

If so, that’s totally fine — I mainly want to understand whether this is the idiomatic expected approach today. (btw, the usingnamespace is deleted from language, so a separate .zig file can not hide the indirection from user, so the codegen must modify the part of file)

Thanks!

2

True.

For the same reason, I don’t think there will be any currently.

Edit: I think manually writing it in this particular scenario is optimal, because the actual requirement here is just to generate a specific structure. I think this approach is not large in scale; string-based metaprogramming is too fragile. If the scale is really unacceptably large, or if there is a very complex need for automatic conversion compatible with multiple versions, code generation at build time may be more recommended.

I’m afraid so; generating at build time is considered the recommended solution.

2 Likes
3

Thanks for the reply! It’s very clear for me.

So there it’s final question: as ‘usingnamespace` is removed, how can I “merge” the generated struct into the unique namespace?

Or is it not an optimal or recommend way in zig? User should use const lua = @import(“Lua.zig”).Lua; instead?

4

From the perspective of the Zig development team, it is a good thing to keep these APIs in a specific namespace rather than in the global namespace.

Additionally, usingnamespace encourages poor namespacing. When declarations are stored in a separate file, that typically means they share something in common which is not shared with the contents of another file. As such, it is likely a very reasonable choice to actually expose the contents of that file via a separate namespace, rather than including them in a more general parent namespace. To put it shortly: namespacing is good, actually.

Although Lua.lua may be somewhat strange, some consideration may be needed in naming the namespace.

1 Like
5

Agreed. But what I want is “merge several namespaces into one and return that one to user”, not “allow user needn’t write namespaces at all”. Maybe the first thing is not a such bad thing?

1 Like
6

I understand this requirement. Sometimes, speaking in general terms can indeed overlook certain specific needs.

However, if you could describe in more detail the specific situations where these namespaces must be generated separately but also need to be reset within the same namespace, we might be able to discuss them together and possibly find a better refactoring solution.

Of course, we also have to admit that, sometimes, there really isn’t a solution that’s very satisfactory; manual forwarding is the most acceptable.

2 Likes
7

By the way, you could alternatively use the @extern builtin and get rid of the APIs namespace altogether. It’s of course arguable whether that style is any better for you than the extern keyword, but I thought I should point it out.

pub const createtable = @extern(
	*fn (L: *State, narr: c_int, nrec: c_int) callconv(.c) void,
	.{ .name = "lua_createtable" },
);
4 Likes
8

You are totally right. So maybe I could manual forwarding for now. As the Lua API still have many details for manual tweak. Thanks for the great explain!

1 Like
9

Good idea! I will change the way bind the Lua API, as a further way, maybe I could make a helper function to do this:

pub const createtable = luaAPI("lua_createtable", "VLii");

Update: I have made this works!

inline fn luaAPI(comptime name: []const u8, comptime sig: []const u8) FnTypeFromSig(sig) {
    return @extern(FnTypeFromSig(sig), .{ .name = name });
}

inline fn FnTypeFromSig(comptime sig: []const u8) type {
    const return_type = getType(sig[0]);
    const param_max = 16;
    var param_buf: [param_max]type = undefined;
    var param_types = std.ArrayList(type).initBuffer(&param_buf);
    std.debug.assert(sig.len < param_max - 1);
    for (sig[1..]) |c| {
        try param_types.appendBounded(getType(c));
    }
    return @Pointer(.one, .{ .@"const" = true }, @Fn(param_types.items, &@splat(.{}), return_type, .{
        .@"callconv" = .c,
    }), null);
}

inline fn getType(comptime c: u8) type {
    return switch (c) {
        'L' => *State,
        'v' => void,
        'i' => c_int,
        's' => [*:0]const u8,
        'f' => *const CFn,
        else => @compileError("Unsupported type"),
    };
}

but In this case, I can not get the type in IDE :confused: zls can not calculate the type of a API, but zig build is passed.

2 Likes
10

I wonder if you can go the other way around — to hand-write public API as Zig (so that both ZLS and a human can immediately see the signatures), and have externs generated.

A sketch:

fn foo(x: u32) void {
    return callLua(.foo, .{x});
}

fn callLua(comptime f_name: anytype, _: anytype) void {
    const f = @field(@This(), @tagName(f_name));

    std.debug.print("{}\n", .{@typeInfo(@TypeOf(f)).@"fn".params.len});
}

Here, you’ll need to repeat argument and function names once, but I think everything else can get reflected upon?

As an aside, from the user-perspective, I think it is important that all signatures are spelled out in a single file, directly. Reading the source code is the most efficient way to grasp the API, and if the API surface itself is “computed”, that becomes much much harder. So, I would start with spelling out the API, and then going for codegen/reflection to actually bind it to lua, rather than the other way around.

5 Likes
11

Good idea! This is just what I do now:

fn getAPI(comptime name: anytype, comptime api: anytype) getAPIType(api) {
    return @extern(getAPIType(api), .{ .name = @tagName(name) });
}

fn getAPIType(comptime f: anytype) type {
    const ft = @typeInfo(@TypeOf(f)).@"fn";
    const params_len = ft.params.len;
    const is_var_args = params_len > 0 and ft.params[params_len - 1].type == null;
    const cf = @Fn(
        &getParamTypes(if (is_var_args) ft.params[0 .. params_len - 1] else ft.params),
        &@splat(.{}),
        ft.return_type.?,
        .{ .@"callconv" = .c, .varargs = is_var_args },
    );
    return @Pointer(.one, .{ .@"const" = true }, cf, null);
}

pub fn getParamTypes(comptime params: []const std.builtin.Type.Fn.Param) [params.len]type {
    var types: [params.len]type = undefined;
    for (params, 0..) |param, i| {
        types[i] = param.type orelse @compileError("unsupported type");
    }
    return types;
}

and the usage:

pub const State = opaque {
    /// Creates a new table and pushes it onto the stack.
    pub inline fn createtable(L: *State, narr: i32, nrec: i32) void {
        getAPI(.lua_createtable, createtable)(L, narr, nrec);
    }

    /// Checks whether the function argument `idx` is a number and returns it as a
    /// `lua.Integer`.
    pub inline fn checkinteger(L: *State, idx: c_int) Integer {
        return getAPI(.luaL_checkinteger, checkinteger)(L, idx);
    }

    /// Raises a Lua error, using the value at the top of the stack as the error.
    pub inline fn raiseError(L: *State) noreturn {
        getAPI(.lua_error, raiseError)(L);
        unreachable;
    }

    /// Raises a Lua error with a formatted message.
    pub inline fn raiseErrorStr(L: *State, fmt: [*:0]const u8, args: anytype) noreturn {
        @call(
            .auto,
            getAPI(.luaL_error, raiseErrorStr),
            .{ L, fmt } ++ args,
        );
        unreachable;
    }
    // ...
};

the API name and extern fn name is not same in some case (for compatible with zlua, e.g. lua_error will use raiseError for name, etc.), so the getAPI should repeat name twice for now.

3 Likes