Packed struct too big / uninitialized

Explain
1

I have expected that packed structs would behave like, for example, packed structs in GCC. I was very surprised to see uninitialized memory read from valgrind referencing io.Writer.writeStructEndian, so I investigated more and it seems that Zig packed structs can be actually way bigger than the sum of its fields, and at the same time, Zig only initializes the fields, not the remaining padding, which makes it unsafe for use in I/O.

For example:

const std = @import("std");

const T = packed struct {
    a0: u32 = 0,
    a4: u32 = 0,
    a8: u32 = 0,
    a12: u32 = 0,
    a16: u32 = 0,
};

pub fn main() void {
    std.debug.print("@bitSizeOf(T)/8 = {}\n", .{@bitSizeOf(T) / 8});
    std.debug.print("@sizeOf(T) = {}\n", .{@sizeOf(T)});
}

This struct should be 20 bytes long, but Zig considers is to be 32 bytes. Is this intended or a bug?

I’m currently resorting to using custom read/write functions using std.mem.asBytes() and then truncating the slice according to @bitSizeOf() to avoid touching the uninitialized bytes, but it still seems like a bug to me.

2

related topic, should help.
also a tip: use extern with align for struct fields instead of packed .

3

see also this post.

4

Ah, ok, so packed struct will be always backed by n^2 bytes. That makes me wonder if writeStruct should not be removed from std, since it’s unsafe with the uninitialized memory and extremely surprising.

5

Yep, this confuses every C programmer.
C’s packed is Zig’s extern with individual align for a field/fields.

Zig’s packed is more like bit-fields, for example:

const std = @import("std");
const PackedStruct = packed struct {
    f1: u3,
    f2: u3,
    f3: u1,
};
pub fn main() void {
    std.debug.print("bitSizeOf(PackedStruct) = {}\n", .{@bitSizeOf(PackedStruct)});
    std.debug.print("sizeOf(PackedStruct) = {}\n", .{@sizeOf(PackedStruct)});
    std.debug.print("alignOf(PackedStruct) = {}\n", .{@alignOf(PackedStruct)});
}

This will print

bitSizeOf(PackedStruct) = 7
sizeOf(PackedStruct) = 1
alignOf(PackedStruct) = 1

Also there was a topic about re-interpreting byte array as a struct and about byte-order conversion.

6

I currently have a PR to change the behavior of the std.io.Reader and Writer to align with your expectations.

7

I also have this PR to improve the language reference description of the memory layout of packed structs

8

Here is a prototype that might be a bit easier:

const std = @import("std");
const native_endian = @import("builtin").target.cpu.arch.endian();

pub fn packFromReader(comptime T: type, reader: anytype, endian: std.builtin.Endian) !T {
    var bytes: [@divExact(@bitSizeOf(T), 8)]u8 = undefined;
    try reader.readNoEof(&bytes);
    if (native_endian != endian) {
        std.mem.reverse(u8, &bytes);
    }
    return @bitCast(bytes);
}

test packFromReader {
    const buf = [3]u8{ 11, 12, 13 };
    var fis = std.io.fixedBufferStream(&buf);
    const reader = fis.reader();
    const PackedStruct = packed struct(u24) { a: u8, b: u8, c: u8 };
    try std.testing.expectEqual(
        PackedStruct{ .a = 11, .b = 12, .c = 13 },
        packFromReader(PackedStruct, reader, .little),
    );
}

and welcome to ziggit!!

9

At first, I was excited about the packed structs in Zig, because if I could have bit-level precision over the individual fields, and have the memory layout stable, I thought could actually use this for internal data files.

Yesterday, after I discovered know the API is really not stable and if this bug is to be fixed, like in the PR you mentioned, the format will be broken, I started working on a static msgpack package, that will be able to serialize structs into a known format that will not change.

Thanks to the compile-time type information, it’s actually quite nice to be able to write these structs into msgpack precisely, without having a second compilation step like with protobuf.

10

I’ve actually been wanting a zig msgpack library for a while!

Something with a similar API / design to msgspec from python (struct = message format).

https://jcristharif.com/msgspec/

11

Yep, I now have a rough library with an API like this:

const Message = struct {
  foo: ?u32,
  bar: bool,
};

const msg = .{ .foo = null, .bar = true, };

var packer = msgpack.packer(&writer, { .struct_format = .array });
packer.writeStruct(Message, msg)

That works very nicely and creates compacts messages that I can read in any language safely.

1 Like