Function pointers, the "export" keyword, and extern structs

Help
,
1

Hello! I’m currently trying to create a custom VFS for SQLite, which requires “subclassing” the vfs object. I’ve ended up with a zig struct that looks like this:

pub const sqlite3_vfs = extern struct {
    iVersion: c_int = 3,
    szOsFile: c_int = @sizeOf(file_types.vfs_file),
    mxPathname: c_int, //max path size
    pNext: ?*anyopaque = null,
    zName: zName_t,
    pAppData: ?*anyopaque = null,
    xOpen: *const @TypeOf(xOpen) = xOpen,
    // etc...
};

Followed by the function definitions, e.g.:

pub export fn xOpen(self: *sqlite3_vfs, filename: zName_t, file: *file_types.sqlite3_file, flags: c_int, out_flags: *c_int) callconv(.c) c_int {
    _ = self;
    _ = filename;
    file.* = file_types.file_instance;
    out_flags.* = flags;
    return SQLITE_OK;
}
// etc...

The vfs is instantiated as a global variable, which gets registered with SQLite:

var vfs_instance = sqlite3_vfs{
    .mxPathname = 200,
    .zName = @constCast("embeddable"),
};

export fn sqlite3_sqliteembeddablevfs_init(_: *anyopaque, _: *anyopaque, pApi: *sqlite3_api.sqlite3_api_routines) c_int {
    if(pApi.vfs_register(&vfs_instance, 1) == SQLITE_OK) {
        return SQLITE_OK_LOAD_PERMANENTLY;
    } else {
        return SQLITE_ERR;
    }
}

This seems to work mostly fine, except that if I use the “export” keyword on another function later in the file (I would also like to export xFullPathname, for instance), earlier functions will be passed completely garbage data.

  • If I only export xOpen, it’s called with the correct arguments
  • If I export xOpen and xFullPathname, xFullPathname will be called correctly, but xOpen gets garbage.
  • If I export a function after xFullPathname, xFullPathname and xOpen both get garbage.

The backtrace looks normal up until the function is actually called.

#0  0x00007ff2dc7e8e35 in main.xFullPathname (self=0x55c28100ed10,
    filename=0x7ffc38922230 "\340\"\2228\374\177", nOut=-1202594056,
    zOut=0x55c280ddafdf <sqlite3OsFullPathname+59> "\311\303UH\211\345H\203\354\020H\211}\370H\211u\360H\203", <incomplete sequence \360>) from ../zig-out/lib/libsqlite_embeddable_vfs.so
#1  0x000055c280ddafdf in sqlite3OsFullPathname (pVfs=0x7ff2dc850640 <main.vfs_instance>,
    zPath=0x55c2b84abdec "scratch.db", nPathOut=201, zPathOut=0x55c2b851f228 "")
    at sqlite3.c:27528
#2  0x000055c280e06a13 in sqlite3PagerOpen (pVfs=0x7ff2dc850640 <main.vfs_instance>,
    ppPager=0x55c2b851be88, zFilename=0x55c2b84abdec "scratch.db", nExtra=136, flags=0,
    vfsFlags=262, xReinit=0x55c280e194c2 <pageReinit>) at sqlite3.c:64408
#3  0x000055c280e19cf6 in sqlite3BtreeOpen (pVfs=0x7ff2dc850640 <main.vfs_instance>,
    zFilename=0x55c2b84abdec "scratch.db", db=0x55c2b851b858, ppBtree=0x55c2b851bb10,
    flags=0, vfsFlags=262) at sqlite3.c:75829
#4  0x000055c280f2df3c in openDatabase (zFilename=0x55c2b851def8 "scratch.db",
    ppDb=0x7ffc38924330, flags=6, zVfs=0x0) at sqlite3.c:190797
#5  0x000055c280f2e2a8 in sqlite3_open_v2 (filename=0x55c2b851def8 "scratch.db",
    ppDb=0x7ffc38924330, flags=6, zVfs=0x0) at sqlite3.c:190920
#6  0x000055c280db6f06 in open_db (p=0x7ffc38924330, openFlags=1) at shell.c:29038
#7  0x000055c280dc914c in do_meta_command (zLine=0x55c2b851fd00 ".open scratch.db",
    p=0x7ffc38924330) at shell.c:34804
#8  0x000055c280dd130e in process_input (p=0x7ffc38924330, zSrc=0x55c280fc62bb "<stdin>")
    at shell.c:37003
#9  0x000055c280dd4669 in main (argc=1, argv=0x7ffc38925a68) at shell.c:38138

Does anyone have any ideas? I don’t have much experience working with the C ABI.

2

Some (equally inexperienced) observations:

  • The sqlite API should have its own *xOpen type you should be able to coerce the function pointer to
  • Since your function has callconv(.c) and the VFS struct has a pointer to it, you might not need to actually export it since it’s never actually being called from C code except through that pointer
  • It also seems like you’re not actually giving the struct the address of the function
1 Like
3

Without exporting, the function is also called incorrectly. It is still in the symbol table and does get called, though, so you’re not wrong.

There seems to be no difference in behaviour when passing &xOpen vs xOpen as the default value to the struct. There also doesn’t seem to be a difference between giving a default value vs setting it when I initialize the struct either.

4

Has OP tried using translate-c to translate interface header files?

I tend to explicitly express interface types. The current approach is to decide the interface type based on the concrete implementation, which feels a bit off.

It looks like it should be pNext: [*c]sqlite3_vfs.

How is zName_t defined? The document states that this parameter may be null, so its definition should be optional or a C pointer.

Why use @constCast? Will it use a value allocated at runtime?

If possible, it might be helpful to show the relevant code using Godbolt.

5

just noting that if @candela would write it manually, you should instead read the code/docs and use one of the other kinds of pointers that better represents how it is used.

[*c] only exists because translate-c cannot know how the pointer is intended to be used since its only source of information is the header.

3 Likes
6

Exact same issue when using the struct generated from translate-c and changing function signatures to match. I’ve never used godbolt before, but I can post that once I figure it out.

7

I tried to reproduce it with the following code:

const std = @import("std");
const sql = @import("sqlite");

// This is what translate C generated:
// pub const sqlite3_vfs = extern struct {
//     iVersion: c_int = 0,
//     szOsFile: c_int = 0,
//     mxPathname: c_int = 0,
//     pNext: [*c]sqlite3_vfs = null,
//     zName: [*c]const u8 = null,
//     pAppData: ?*anyopaque = null,
//     xOpen: ?*const fn ([*c]sqlite3_vfs, zName: sqlite3_filename, [*c]sqlite3_file, flags: c_int, pOutFlags: [*c]c_int) callconv(.c) c_int = null,
//     xDelete: ?*const fn ([*c]sqlite3_vfs, zName: [*c]const u8, syncDir: c_int) callconv(.c) c_int = null,
//     xAccess: ?*const fn ([*c]sqlite3_vfs, zName: [*c]const u8, flags: c_int, pResOut: [*c]c_int) callconv(.c) c_int = null,
//     xFullPathname: ?*const fn ([*c]sqlite3_vfs, zName: [*c]const u8, nOut: c_int, zOut: [*c]u8) callconv(.c) c_int = null,
//     xDlOpen: ?*const fn ([*c]sqlite3_vfs, zFilename: [*c]const u8) callconv(.c) ?*anyopaque = null,
//     xDlError: ?*const fn ([*c]sqlite3_vfs, nByte: c_int, zErrMsg: [*c]u8) callconv(.c) void = null,
//     xDlSym: ?*const fn ([*c]sqlite3_vfs, ?*anyopaque, zSymbol: [*c]const u8) callconv(.c) ?*const fn () callconv(.c) void = null,
//     xDlClose: ?*const fn ([*c]sqlite3_vfs, ?*anyopaque) callconv(.c) void = null,
//     xRandomness: ?*const fn ([*c]sqlite3_vfs, nByte: c_int, zOut: [*c]u8) callconv(.c) c_int = null,
//     xSleep: ?*const fn ([*c]sqlite3_vfs, microseconds: c_int) callconv(.c) c_int = null,
//     xCurrentTime: ?*const fn ([*c]sqlite3_vfs, [*c]f64) callconv(.c) c_int = null,
//     xGetLastError: ?*const fn ([*c]sqlite3_vfs, c_int, [*c]u8) callconv(.c) c_int = null,
//     xCurrentTimeInt64: ?*const fn ([*c]sqlite3_vfs, [*c]sqlite3_int64) callconv(.c) c_int = null,
//     xSetSystemCall: ?*const fn ([*c]sqlite3_vfs, zName: [*c]const u8, sqlite3_syscall_ptr) callconv(.c) c_int = null,
//     xGetSystemCall: ?*const fn ([*c]sqlite3_vfs, zName: [*c]const u8) callconv(.c) sqlite3_syscall_ptr = null,
//     xNextSystemCall: ?*const fn ([*c]sqlite3_vfs, zName: [*c]const u8) callconv(.c) [*c]const u8 = null,
// }

var vfs = sql.sqlite3_vfs {
    .iVersion = 3,
    .szOsFile = @sizeOf(sql.sqlite3_file),
    .mxPathname = 200,
    .pNext = null,
    .zName = "embeddable",
    .pAppData = null,
    .xOpen = xOpen,
    .xDelete = null,
    .xAccess = null,
    .xFullPathname = xFullPathname,
    .xDlOpen = null,
    .xDlError = null,
    .xDlSym = null,
    .xDlClose = null,
    .xRandomness = null,
    .xSleep = null,
    .xCurrentTime = null,
    .xGetLastError = null,
    .xCurrentTimeInt64 = null,
    .xSetSystemCall = null,
    .xGetSystemCall = null,
    .xNextSystemCall = null,
};

pub fn xOpen(self: [*c]sql.sqlite3_vfs, zName: sql.sqlite3_filename, file: [*c]sql.sqlite3_file, flags: c_int, pOutFlags: [*c]c_int) callconv(.c) c_int {
    _ = self;
    _ = file;
    _ = flags;
    _ = pOutFlags;
    std.debug.print("xOpen called: {s}\n", .{zName});
    return sql.SQLITE_ERROR;
}

pub fn xFullPathname(self: [*c]sql.sqlite3_vfs, zName: [*c]const u8, nOut: c_int, zOut: [*c]u8) callconv(.c) c_int {
    _ = self;
    std.debug.print("xFullPathname called: {s} (nOut = {})\n", .{zName, nOut});
    @memcpy(zOut[0..4], "abc\x00");
    return sql.SQLITE_OK;
}

pub fn main() !void {
    if (sql.sqlite3_vfs_register(&vfs, 1) != sql.SQLITE_OK) {
        return error.sqlite_vfs;
    }
    std.debug.print("Registered\n", .{});
    
    var db: ?*sql.sqlite3 = undefined;
    if (sql.sqlite3_open("test.db", &db) != sql.SQLITE_OK) {
        return error.sqlite_open;
    }
    
    if (sql.sqlite3_close(db) != sql.SQLITE_OK) {
        return error.sqlite_close;
    }
}
build.zig 
const std = @import("std");


pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});
    
    const sqlite_c = b.addTranslateC(.{
        .root_source_file = .{.cwd_relative = "/usr/include/sqlite3.h"},
        .target = target,
        .optimize = optimize,
    });
    const sqlite = sqlite_c.createModule();
    sqlite.linkSystemLibrary("sqlite3", .{});
    
    const exe = b.addExecutable(.{
        .name = "sqlite",
        .root_module = b.createModule(.{
            .root_source_file = b.path("main.zig"),
            .target = target,
            .optimize = optimize,
            .imports = &.{
                .{ .name = "sqlite", .module = sqlite },
            },
        }),
    });

    b.installArtifact(exe);
    
    const run_step = b.step("run", "Run the app");
    const run_cmd = b.addRunArtifact(exe);
    run_step.dependOn(&run_cmd.step);
    run_cmd.step.dependOn(b.getInstallStep());
    if (b.args) |args| {
        run_cmd.addArgs(args);
    }
}

But this correctly prints:

Registered
xFullPathname called: test.db (nOut = 201)
xOpen called: abc

before exiting with the sqlite_open error.

Is it possible for you to create a minimal reproducing (counter)example?

Also: try to compile with -fllvm (or .use_llvm = true in build.zig) to test if it also happens when using the LLVM backend.

1 Like
8

I think my debugger may have been leading me astray. Your example works… but adding the print statement to mine also works fine, lol. In GDB the memory looks corrupted:

Breakpoint 2, 0x00007f82a93ea6a5 in main.xFullPathname (self=0x560ec7bc0698, filename=0x7fff6f6f68c0 "@ioo\377\177", nOut=1,
    zOut=0x560eaad884b0 "\205\300\017\205\270\a") from zig-out/lib/libsqlite_embeddable_vfs.so
(gdb) continue
Continuing.

Breakpoint 1, 0x00007f82a93ea585 in main.xOpen (self=0x560ec7bc0698, filename=0x7fff6f6f68c0 "@ioo\377\177", file=..., flags=-943978280, out_flags=0x560ec7bc0944)
   from zig-out/lib/libsqlite_embeddable_vfs.so

LLDB thinks xFullPathname is being passed entirely null pointers, which is definitely wrong:

* thread #1, name = 'sqlite3', stop reason = breakpoint 2.1
    frame #0: 0x00007fbbc28f7bbf libsqlite_embeddable_vfs.so`main.xFullPathname(self=0x0000000000000000, filename=0x0000000000000000, nOut=0, zOut=0x0000000000000000) at main.zig:123:121
   120 	        _ = pResOut;
   121 	        return SQLITE_ERR;
   122 	    }
-> 123 	    pub fn xFullPathname(self: [*c]sqlite3_vfs, filename: zName_const, nOut: c_int, zOut: zName_mut) callconv(.c) c_int {
    	                                                                                                                        ^
   124 	        std.debug.print("xFullPathname called: {s}\n", .{filename});
   125 	        _ = self;
   126 	        //temp stub
(lldb) v
(main.struct_sqlite3_vfs *) self = NULL
(unsigned char *) filename = 0x0000000000000000
(int) nOut = 0
(unsigned char *) zOut = 0x0000000000000000
([:0]const u8) wLength = (ptr = 0x0000000000000000, len = 0)

So it looks like I may have been chasing a red herring. I do still get a logic error, but I think it’s more likely due to not fully setting up the vfs_file struct.