Incorrect alignment from sqlite3_aggregate_context

:wave:

I’m trying to write an SQLite3 extension in Zig, and in particular, to convert this bit of code from C:

typedef struct duration_state
{
  double total;
  double prev;
} duration_state;

static void duration_step(sqlite3_context *ctx, int argc, sqlite3_value **argv)
{
  duration_state *state = (duration_state *)sqlite3_aggregate_context(ctx, sizeof(duration_state));
  if (state == NULL)
    return sqlite3_result_error_nomem(ctx);

  double time = sqlite3_value_double(argv[0]);
  double diff = time - state->prev;
  if (diff < 300)
    state->total += diff;
  state->prev = time;
}

the problematic line is

duration_state *state = (duration_state *)sqlite3_aggregate_context(ctx, sizeof(duration_state));

it calls sqlite3_aggregate_context and it seems to work fine in C and Zig’s -O ReleaseFast but fails the runtime check in Zig’s -O ReleaseSafe.

Here’s what I’ve tried in Zig:

const DurationAggState = struct {
    sum: f64,
    prev: f64,
    pub fn add(self: *DurationAggState, time: f64) void {
        const diff = time - self.prev;
        if (diff < 300) self.sum += diff;
        self.prev = time;
    }
};

pub fn durationStep(ctx: ?*c.sqlite3_context, argc: c_int, argv: [*c]?*c.sqlite3_value) callconv(.C) void {
    _ = argc;

    const state = @ptrCast(
        ?*DurationAggState,
        @alignCast(
            @alignOf(DurationAggState),
            sqlite3_api.aggregate_context.?(ctx, @sizeOf(DurationAggState)),
        ),
    );

    if (state == null) return sqlite3_api.result_error_nomem.?(ctx);
    const time = sqlite3_api.value_double.?(argv[0]);
    state.?.add(time);
}

where the problematic line has become

    const state = @ptrCast(
        ?*DurationAggState,
        @alignCast(
            @alignOf(DurationAggState),
            sqlite3_api.aggregate_context.?(ctx, @sizeOf(DurationAggState)),
        ),
    );

The error I’m getting is

thread 8652281 panic: incorrect alignment
  /.../src/main.zig:86:44: 0x109928f73 in durationStep (main)
    sqlite3_api.aggregate_context.?(ctx, @sizeOf(DurationAggState)),
                                   ^
                                   Panicked during a panic. Aborting.

What am I doing wrong with the alignment?

At first sight everything seems fine. One suggestion that I have is to maybe use print debugging to test if the pointer to your DurationAggState is the same when you give it to sqlite (assuming that’s how this works) and when you take it out.

Another question is: how are you allocating these values? I kinda doubt that’s the problem, but the allocator that you’re using might simply ignore any alignment requirement and give you a misaligned pointer from the start. What happens if you immediately @alignCast the pointer that the allocator returns you?

1 Like

how are you allocating these values?

I think sqlite3_aggregate_context allocates the memory:

The first time the sqlite3_aggregate_context(C,N) routine is called for a particular aggregate function, SQLite allocates N bytes of memory, zeroes out that memory, and returns a pointer to the new memory.

So I’m not using any std.mem.Allocator right now.

But I’m very new to both Zig and C so I might be completely misunderstanding what you are asking :slight_smile:

I’ll try the two suggestions and report back. Thank you for the help!

Makes sense, in hindsight I could have assumed as much since it takes in a size.
well, here’s the problem: the pointer that you get back might or might not have the alignment expected by your struct. This is one of the many footguns that C has since there is no easy language feature to work with alignment.

For clarity: alignment is a requirement that types have (structs, numbers, pointers, etc) and that can go out of whack when doing dynamic allocations. For example if a struct has an aligment of 8, then it can only live at a pointer address that is divisible by 8. The alignment of a struct depends on its fields and so with what sqlite3 gives you, there is no clean way to ensure this. It’s not uncommon for example for C allocators to always return word-aligned memory, since that tends to be the maximum alignment that a type might need.

One thing that you could do is to “waste” a bit of memory that sqlite gives you by aligning forward the pointer that you get. std.mem.alignForward does this. Of course if you allocate just the right amount of memory to hold your struct, then throwing away bytes will cause issues as you won’t have enough space for your struct. To fix this you just need to ask for a bit more memory.

So in pseudo code you could do this:

const size = @sizeOf(DurationAggState);
const alignment = @alignOf(DurationAggState);
const unaligned_ptr = sqlite3_aggregate_context(size + alignment);
const ptr = std.mem.alignForward(unaligned_ptr, alignment);

Note that one downside of doing this is that you will have to keep track of unaligned_ptr, since that 's the pointer that you will have to communicate to sqlite to free the memory. Up to you if you
want to keep track of both pointers or if you want to recompute the aligned pointer on the fly.

I personally would recommend the second option. For convenience you could create a fromRawPtr function in DurationAggState that does the pointer cast and forward align for you.

Let me know if it works!

2 Likes

I’ve added a print with pointer, and it shows the address is 102a7acdc which is 4339510492 in base10 and it’s not not divisible by 8, so I guess that’s the problem. I’ll now try alignForward.

std.mem.alignForward worked, thank you!

And I found that the issue doesn’t seem to be in sqlite+zig interop but rather there is something happening when erlang is added to the mix where erlang calls an sqlite nif which calls a zig function.

Here’s a repo for anyone who wants to reproduce the issue:

And the alignForward workaround:

Note that the workaround doesn’t seem to be required when using sqlite3-cli as the ctx pointer is properly (8-byte) aligned.

It’s not uncommon for C allocators to not care about alignment. This has the unfortunate consequence that it makes harder to port the code away from x86-64 (x86 tends to not have problems with weirdly aligned data). Arm for example is much more picky with alignment.

Zig is just forcing you to be more precise with this stuff.

1 Like

forgot to reply to this: erlang is most probably linking its own allocator into sqlite, that’s why you see a different behavior

1 Like

Right, that’s what I’m going to explore in the next few days! Thank you again!