I can’t figure out how to do some things in Zig that I do in C.
In many language interpreters for example, there is a small header at the front of every object, followed by unspecified other data. The header should be at the top, so that for any object, you can get its vtable, for example, or a list link and flags for the garbage collector.
Zig doesn’t guarantee the order of struct fields, unless you use packed, but packed structs cannot hold other structs like an ArrayList, etc. And I would just need order, not packing. So how can I create objects whose first bytes are always a header followed by some other struct? Thank you.
Are you referring to extern structs? Documentation - The Zig Programming Language
I saw you mentioned packed struct, but I want to be clear.
An extern struct has in-memory layout guaranteed to match the C ABI for the target.
Also, welcome to the forum!
Well, I don’t need to match a C ABI. I just tried packed because it guaranteed order. And extern structs cannot contain regular structs either, so I can’t put an ArrayList in an extern struct.
An error occurred:
playground/playground3935810399/play.zig:15:33: error: extern structs cannot contain fields of type ‘hash_map.HashMapUnmanaged(play.V,play.V,hash_map.AutoContext(play.V),80)’
playground/playground3935810399/play.zig:15:33: note: only extern structs and ABI sized packed structs are extern compatible
I know it’s kinda janky, but you can copy the code for the HashMapUnmanaged or whatever other container you want and tack on the keyword extern or packed in front of it, and I think it should work.
Can I count that as code reuse?
1 Like
Hmm, is there some way at comptime I can take a struct type and tack on an extern tag?
So, I tried this, don’t know quite what I’m doing, and got an error.
pub fn externalize(comptime T: type) type {
var outInfo = @typeInfo(T);
switch (outInfo) {
.Struct => |*info| info.*.layout = std.builtin.Type.ContainerLayout.Extern,
.Union => |*info| info.*.layout = std.builtin.Type.ContainerLayout.Extern,
else => @compileError("expected struct or union type, found '" ++ @typeName(T) ++ "'"),
}
return @Type(outInfo);
}
An error occurred:
playground/playground1071368380/play.zig:20:12: error: reified structs must have no decls
full source:
const std = @import("std");
const builtin = @import("builtin");
const V = union(enum) {
o: *Obj,
i: i64,
f: f64,
u: u64,
b: bool,
nil,
};
pub fn externalize(comptime T: type) type {
var outInfo = @typeInfo(T);
switch (outInfo) {
.Struct => |*info| info.*.layout = std.builtin.Type.ContainerLayout.Extern,
.Union => |*info| info.*.layout = std.builtin.Type.ContainerLayout.Extern,
else => @compileError("expected struct or union type, found '" ++ @typeName(T) ++ "'"),
}
return @Type(outInfo);
}
const TableObj = extern struct {
obj: Obj,
table: externalize(std.AutoHashMapUnmanaged(V, V)),
};
const Obj = extern struct {
nextObj: ?*Obj = null,
typeTag: *Obj,
format: u8,
gccolor: u8,
};
pub fn main() void {
std.debug.print("Hello, {s}! (using Zig version: {})\n", .{"world", builtin.zig_version});
std.debug.print("{d} {d} {d}\n", .{@sizeOf(V), @sizeOf(Obj), @sizeOf(TableObj)});
}
@typeInfo(comptime T: type) std.builtin.Type
@Type of @typeInfo is not what you want here. I’ll do some digging and see what options you have.
It sounds to me like you’re trying to retrieve a field from an object based on it’s alignment (given that in C, the first field will be upfront). Am I reading your use case correctly here?
I don’t care about working with C. What I want is an intrusive data structure at the top of blocks of memory, with a varying payload after. The header must always be at the top, and the payloads may be unknown to the code dealing with the header.
Alright (and the C analogy was just as an example, I get that you don’t necessarily want to mimic C). Let me look and see what I can dig up.
Can you give us an example of how you want to use this code - intrusive structures can have a lot of use cases (some are used to link structures together, for instance).
And actually, it looks like you want to have an object pointer at the top of your struct and you want to point to that field in another object.
Have you looked into @fieldParentPtr? There’s some code that uses this in conjunction with allocators to identify generic objects in thread pools (they use the intrusive node to walk back up to the structure itself from a field in the parent structure, you can find that example in the thread pool implementation in the standard library).
1 Like
Reified structs are structs built with @Type. They can’t have declarations, like functions or static variables. Its quite a limitation… You’d have to actually copy-paste the code to put the keyword.
Unless I’m missing something, there’s an example of this sort of thing in the GeneralPurposeAllocator:
here’s how it’s allocated:
and here’s how it’s freed:
There are accessor functions in BucketHeader that calculate the offset to particular trailing sections, like this:
If you make the Header struct extern then you’d get guaranteed field order, but the idea would be the same: allocate @sizeOf(Header) + trailing_size bytes with alignment @alignOf(Header), cast it to *Header, and then make sure that you can still free the entire allocated slice (so either store the size of the trailing data in the header, or make it so the size of the trailing data can be calculated when it’s time to free).
EDIT: Also worth looking at how the bucket size is calculated to see how it deals with ensuring alignment of the trailing data:
1 Like
I understand squeek502’s answer, which is to do memory layout by hand. I’m not sure I understand how to use @fieldParentPtr. So, is the following correct?: the header can be anywhere within the struct. Say a caller, who only knows the type of the header and not the type of the memory block, calls a function through the vtable pointer in the header. That function, which does know the type, uses @fieldParentPtr to translate the header pointer to the pointer to the containing type. yes?
… and what is the cost of @fieldParentPtr? just a pointer addition?
Yes you have it right.
The cost is a pointer subtraction with an immediate operand. The compiler is also permitted to choose struct field order, so it may notice this pattern being used and decide to put the base struct first. Such an optimization is not currently implemented, however.
I hate this for a system language. I often have specific field orders to decrease working cache line set or allow for fast forwrading over structs as I run down a collection of them. Or I have padding specifically to split values over cache lines but still not have a ton of cache lines with a single value on them. I don’t want the compiler screwing with that. And remember you dont always want the item accessed the most to bethe fastest - there are very valid use cases where 1 in 100 times you take a certain code path or access a certain element and that one time is the one you want to as fast as possible - all the other iterations are just waiting on that one. So just optimizing by most accessed isn’t going to do it.
@jnordwick You can opt-out of this behavior by using an extern struct (or packed struct, though that usually has different use cases).
Yes, but then you cannot contain any normal struct (e.g. ArrayList) within a packed or extern struct.
Is that really a good idea? If it’s outside of your code, then you cannot control it’s memory layout.
As soon as the library authors start changing stuff you are screwed.
Maybe one day someone from Zig decides to change the way Allocators are stored in memory.
If you used an ArrayList then your struct changes with the Allocator and you’ll spend the entire day figuring out why your code suddenly doesn’t work anymore.
I think a better solution would be to make an extern struct for all the fields you need to control the memory layout of, and embed that together with the ArrayList and whatever into a regular struct.
1 Like