I’ve documented the status quo here:
Run zig build to verify.
4 Likes
This is the essence of why the improving the status quo as discussed here is a difficult ask.
In brief: the C ABI is a lie. The C standard does not mandate an ABI, but due to the prevalence of dynamic linking in C, the various operating systems and compilers converge around a strict convention for code which dynamically links with other code.
This is an issue because, let’s say Zig defined its own enriched ABI. There’s no way for that object code to advertise that it’s laid out differently from what the C consensus demands, because our object formats lack any such facility, unless you want to count symbol mangling, which I don’t.
C++ solves this problem by creating enormous problems for everyone, Zig solves it, like most other languages, by punting to the least-common-denominator.
Mostly, Zig operates on a closed-world assumption, and mostly, that works for what Zig is good at. My one ask would be to define a layout for slices already, an official C struct consisting of the pointer first and the length second. There’s really no reason not to do this.
But to do anything more sophisticated requires work at the operating system and object code format level which no one seems to be interested in. The freedom Zig gains with no-defined-layout guarantee is mostly about calling convention, rather than the ins and outs of how data formats themselves are laid out and evolve.
My point being, dynamic linkers fly completely blind. If object code is going to play in that arena, it needs to either advertise how it works, which calls for a new object format basically, or it needs to do like C++ and cause problems, or it needs to emit interfaces in a C compatible manner. One of these is very challenging, the other is rude, so that leaves door #3.
Cursed, maybe, hack, definitely, but I’m not convinced it needs improvement (and filed an issue expressing my hope that this behavior would be guaranteed going forward). Use of @setRuntimeSafety clearly expresses what’s going on there.
2 Likes
Honestly I think people are overthinking the problem. The only real problem is that Zig likes to not being forced into a locked memory layout for objects, yet dynamic linking of objects requires two binaries to agree on memory layout. One could argue that anytype and comptime is another but clearly by definition they cannot be allowed to cross dynamic boundaries as a dynamic boundary is runtime by definition.
In practice when compiling we need a way to tell the compiler that a given source zig file are replacing the struct known as MySuperUtf8String switching it to be a packed struct looking like this. It can then attempt to compile the code and fail if there are missing fields, mismatched types or what not. Obviously one could hand edit these, but let’s no go editing std files with packed structs just to support dynamic.
Similarly we need the ability to generate packed structs as an output from our build pipeline. We can’t do this comptime as comptime is not allowed to have side effects so we have to do this from build.zig through some sort of observer.
This way if people want a static API they can agree on a packed struct Convention and build both binaries with it. Quite possibly a convention they asked zig to deliver on their first release. However if they want optimal performance in their main app and can accept rebuilding plugins for each version they can have an API that changes with each build.
That still leaves slice as the odd ball but as you said that’s really just a struct.
Once this is done, the rest is just a library in std.
For my use cases where I want to add plugin support into a larger system I would need a library that allows me to export structs and functions from the host. But this is just vtables to be filled out when loading the module.
One could also imagine someone making a closed source library where they leave an api for you, complete with a loadmodule method that grabs all the addresses for the vtables it needs from your system and hooks it into their library. This would keep their binary size down and ensure that you don’t have two dynamic libraries that ends up conflicting on whether the pointer or the length is the first member of a slice. 
About how other languages will support our packed struct that’s really not my concern. It’s possible but will take effort.
This is not a small thing. But I don’t see a real way around it. Until we have a clean way to replace certain parts of code base with dynamic replacements (generated packed structs with vtables) dynamic linking zig dlls will be one massive hack.
1 Like
Once you have dynamic libraries around, you have to know exactly how they’re laid out. They can sit and ferment for years and people are going to want them to still link. C solves this problem by being very old and set in its ways. C++ solves this problem poorly.
It’s not about Zig defining a layout, it’s about that layout ossifying and never being changeable again. That’s ok for C because C never changes, but is it? Is it ok for C? Why is int still 32 bits again? ABI. ABI is why.
The solution to this problem is metadata in the object code. It’s a hard problem, mostly for social reasons. But reinventing the severe (!) problems C++ has had, which are caused by ABI compatibility, is not a solution to it, and pretending that, right now, we know what Zig object code should look like in five years, is also not a solution to it.
Better than under-thinking it.
As I’ve said elsewhere on this board, the ability to dynamically link with Zig-native object code is a good long-term goal for the language. But I think you’re discounting the problems which would come with just YOLOing out an 80% solution here.
1 Like
Need I remind you that the advantages of not yet being version 1 is that we’re allowed to break a few eggs? It is true that you only live once, but it doesn’t kill you to make a mistake, fail fast is a proven way to achieve your goals.
What I suggested is to support structs, functions and slices. Whatever solution we end up choosing I bet it involves structs, functions and slices.
When I look at how .NET supports dynamic linking(to great success I might add) it involves a ton of versioning rules and hidden magic, exactly the kind of thing that the zig language is actively trying to avoid.
The language features should be aimed towards what is needed, eg. sharing base features so that an array list can be passed across the dynamic border and both sides can add elements, remove elements, sort the list, etc. What I proposed is a simple way to achieve this that doesn’t require the entire foundation of the language to be replaced.
Obviously I only dealt with 80%, why would I start rambling about the troubles that our global errors concept is going to produce? It’s a problem, but it can be fixed. I find this concept that everything has to be the perfect solution before it can be said out loud counter productive when we have a large body of developers with a lot of valuable experience to draw on.
3 Likes
I don’t mind saying things out loud at all, but I think at this time it would be premature. ABI breaks fail silently and corrupt data, API breaks fail loudly and, mostly, just refuse to compile. We should stick with the latter for now.
1 Like
The best way to deal with this issue is through function transform. We should have something in the standard library that transform Zig functions to C functions and back. We would transform a fn (const u8) void to a fn ([*]const u8, usize) void on export and transform it back on import.
2 Likes
This part of your statement I agree with. But I do not consider it premature at all, on the contrary I think we should get the work started to avoid being forced to rush in something half baked when suddenly the zig project is hit by the bus factor and popular demand calls for version 1.0.0.
None of these risks really matter, until zig reaches version 1.0.0 one cannot expect the ABI to be compatible across versions. That is no different from many other languages. By adding a single core feature we can start building the different flavors of dynamic linking.
Even though the dynamic monolith of .NET often have 50+ dlls and is very far from the core ideal of zig, it does serve a purpose with clear benefits, there is currently no language without garbage collectors serving this audience.
Personally I’m interested in the plugins/extensions that add functionality to the core program with a clear contract.
Still others will want to wrap up their code in libraries and ship them in compiled form.
All of these use cases can be tested and vetted by adding one feature. The ability for the linker to swap one struct or function for another.
1 Like
Do you see a reason for this advertising to be done? Changing the layout would be an ABI change that would warrant a major version bump, right?
You’re very right. But this is why I think the ability to specify a layout explicitly is important. That way, Zig can still do whatever layout optimizations it wants to do for non-annotated structs, and the user can require a specfic, particular layout. This could be done with comptime today… if @Type could reify decls.
I’m… confused, about why you think this is necessary.
Please excuse my ignorance, but isn’t this is a complete non-issue if control over the layout is handed to the language user? A change in the first parameter to @abiLayout, or however one would call it, would be an ABI-incompatible change. Like reordering some struct fields, requiring a major version change. C++ uses symbol mangling, leading to many issues. Zig does not need this in my opinion and I don’t think this proposal benefits from it either. #3 can still be done for C interop.
1 Like
It’s not any different than any other non-C language though (e.g. C++ or Rust don’t have a stable ABI either, nor do they guarantee a memory layout for types - for instance you simply can’t rely on a specific memory layout for a C++ class with virtual methods or even a derived class without virtual methods, or what a Rust Option, Result or Enum looks like in memory, especially since the compiler may use tricks like using the nullptr value for representing the ‘non-value’ state.
And tbh I don’t think the problem is worth solving - tunneling high-level-language concepts through restricted C ABI shims is a well-understood problem and has been time and time again solved via 3rd-party tooling, and when trying to do it right you might easily end up in an engineering quagmire like the WASM component model: https://component-model.bytecodealliance.org/
3 Likes
I thought about linking to this actually. Quagmire or not, it’s the best example of the kind of effort which would need to go in to defining a, well, let’s call it a Common Object Model, shall we? An enriched ABI which has some hope of not constantly breaking, of allowing interoperable dynamic libraries between multiple languages without using C.
Because really, there’s no point in doing it other than to do it right. With status quo, you define an export interface, and your Zig lib can link with everything. Accepting all the disadvantages, and hard work, of building a Zig-specific linking model, and then only being able to link with Zig (and not Swift, or C++, or Rust) would be retrograde. Zig is about reusable software, after all, so making it less reusable does not feel like it’s stepping in the right direction.
What I’m gesturing at is achievable on a technical basis, no question. I think it’s just short of sociopolitically impossible though. As you point out, everyone who words on code in this space has experience smuggling data and language constructs through the C layer, and while it does suck, it’s really not that bad. It’s the kind of local optimum which is hard to break out of.
2 Likes
I think Zig could potentially become the single dependency that serves as base layer for implementing different languages, with that in mind I think it could become an alternative/competitor to the family of llvm-based languages and with that, it should be a lot easier to have a bigger set of common data-structures and layouts that are supported by these language implementations across that language boundary.
I think I am possibly more interested in from source compilation compatibility across different languages, instead of object format artifact compatibility.
Basically if it was easy to compile a big polyglot program completely from source without having to worry about the details, because they all were automated behind a unified build system.
There would still be people who want to ship compiled artefacts, but I think having seamless integration, when compiling everything from source, could be a huge workflow improvement, for things that can be distributed as from source only.
I think the python libs cross-compiled via zig compiler was already a nice step in that direction and I would like to see more along those lines. (While binary workflows are also good, personally I care more about from source and better dev workflows)
4 Likes
Except … it’s not a well-understood problem?
There is no getting around having a “description” somewhere. For C, that’s the “.h” file. And “.h” files are a notoriously crap language for describing things that don’t exist in C. For example, describing slices is a nightmare.
The WASM Component Model has WIT. That’s the description. You can’t avoid that.
Now, I have beefs with the WASM Component Model. However, it is not obvious what would be a better tradeoff. And a bunch of the complexity of the WASM Component Model is how you deal with pulling in the component as part of an unreliable supply chain. And they leave the memory layout to raising/lowering that seems to require a runtime–I don’t like that very much.
Yes, the WASM Component Model feels very “Rust-centric” to me. However, that is simply due to the fact that “Rust folks” are the ones engaged and doing the work and code. If you want more “Zig-centric”, then some Zig folks are going to have to tramp over there, engage with that community and sling some code.
2 Likes
What I’d really like to see in all language compilers is a ‘API JSON dump’ feature, which exports the public API of a module into an easy to parse JSON format suitable for code-generating language binding shims (or even transform into another IDL like WIT). It doesn’t even have to be a language agnostic standard format, it just needs to be easy to parse.
E.g. for generating language bindings for the sokol headers I use clang’s ast-dump feature, but this is way too noisy and requires to extract the actually relevant data into a simplified format. This is similar to shader language compilers dumping shader interface reflection data (e.g. Slang’s --reflection-json feature: slang/docs/command-line-slangc-reference.md at master · shader-slang/slang · GitHub)
I’m not a fan of maintaining IDL files manually, the module source code of the source language must be the single source of truth.
Yes, the WASM Component Model feels very “Rust-centric” to me.
Exactly, baking high level types like List, Option, Result, Tuples, Variant etc… is a stupid decision. Why exactly these types and not others, where do you draw the line…
…all of this is useless without fixed and defined memory layouts for types though, and I think Zig’s extern struct approach is the way to go even though it only works for the ‘C compatible subset’ of the type system.
1 Like
For WASM Components, maybe not so stupid. It’s worth pointing out that Zig has acceptable equivalents to all of these, which for the WASM target makes them a reasonable way to emit those types.
I did get the impression that WASM Components is really just Rust with extra steps, to an almost comical degree. Imagine what it will look like if the C++ gang ever shows up 
For a ‘futuristic ABI’ project, more is needed: it would need to be able to specify both general and particular data. Meaning that the C++ crew (or Carbon, whatever) wouldn’t have to muscle in on the committee to get a piece of the action, they’d be able to define their layout in a metaformat, which other language compilers could read, and create an object-code shim allowing that language’s semantics to work with all that C++ weirdness.
Technically, it’s really not so weird a project, not that different from DWARF if you squint. But getting all the stakeholders together, and having any prayer of a design-by-committee process producing a quality specification, that beggars my considerable imagination.
But maybe Zig could Just Do It and present it as a fait accompli. Probably not soon though.
1 Like
Sounds like a use for the CTF debug format that dtrace used to access structs and function parameters.
1 Like