For context I removed the comptime query checking code in favor of leveraging the error reporting that I get from sqlite by preparing all queries at application startup. In my case it’s the most practical approach, but I do believe that build-time checking (i.e. not comptime, but build time) can be valuable. Unfortunately it requires a lot of effort to get done and with sqlite you want to prepare your statements anyway.
3 Likes
It would be nice. But I think a challenge in this is the syntax used to describe permissible types.
In general we’re looking for a way to specify a predicate on types, using some restricted syntax.
This gets complicated quicky, especially if you want to infer type parameters, like here:
fn find(seq: SequenceTrait(T), needle: T)
The current “solution” is more accommodating and simple: Have the full Zig language to check type constraints, and note the requirements in the function doc.
It would be nice to put some kinds of constraints in syntax in the function prototype (and compile-time interfaces), but I think it will get complicated real quick.
1 Like
I have similar code in Zig to parse Ethernet frames. I find the main issue is that Zig doesn’t have big enendian types, so it’s user responsibility to out the right cast at the right places
(Apparently proposal is accepted Endian-aware integer types · Issue #3380 · ziglang/zig · GitHub
I treat packets as pointer to struct where the struct is the header , and then provide method to access the variable length part of the packet.
It would be cool if Zig had a custom union type where you could specify the bitoffset of the union tag, but ultimately it works well with a payload() method on the Eth struct that skips the vlan tags.
If you’re intereted I’ll try to open source the code
3 Likes
There is a related proposal for this feature:“Consider if @src() in an inline function should refer to the caller’s location”.
For the compiler source to be documented and extensible.
I just realized that packed/extern structs are not quite as powerful as bit strings. Consider Fragment Offset field from IP header: IPv4 - Wikipedia
If you want to represent as packed struct on a little-endian machine, I think you’ll have to
const IPv4Header = packed struct {
fragment_offset_hi: u5,
flags: u3,
fragment_offset_lo: u8,
};
4 Likes
Interesting, that is very similar to my first ever programme written in Zig over the last couple of weeks. In my case I am reimplementing an existing C# programme that does the same thing, partly for the learning experience and partly because it is much quicker and has better cross-platform support
Anyhow, back on topic
Thus far, other than having to cook up a replacement for “mktime”, my only real bugbear, and so my token suggestion for this topic, is that I would like to be able to explicitly tag fields in a structure as private (or invisible outside the current file if you prefer) so that I can keep some private internals of my “context” structure to pass around thus in place of non-strict workarounds
A simple cat with simple needs
Hot reloading/binary patching built into the compiler.
4 Likes
Just curious, isn’t hot loading a runtime and loader requirement?
What do you mean by runtime in this case?
FYI: this is what I’m thinking of hot code swapping · Issue #68 · ziglang/zig · GitHub
I see; it was an old issue. Thanks for the link.
By runtime requirement, I meant the loading of modified code and patching of existing running code needs to be done by some runtime code in the target process. This “runtime code” is a separate hot-loading library. The Zig compiler can produce code suitable for runtime injection; it is unlikely to magically reach into a running process and modify its code.
Actually most of the stuffs are almost there. Zig compiler currently can produce dynamic library (.dll and .so), and it produces the debug info in .pdb file on Windows and (DWARF debug info in .so file on Linux?). The debug info maps the relative virtual address of each function and its name. A new .pdb file is produced after a compilation of a modified function.
To hot load a function, the hot-loading library looks at the old .pdb to find out the function’s old relative virtual address and figure the absolute address of the function in the running process (loaded module’s base address + relative virtual address). It then looks in the new .pdb to find the function’s new relative virtual address in the new .dll/.so. It loads the new .dll/.so and figures out the new absolute address of the function (the new loaded module’s base address + relative address). Finally it patches the old address of the function to jump to the new absolute address.
For patching, Windows has the Detours and MinHook libraries; Linux has the patchmem library. There might be others.
For triggering hot loading, the hot-loading library can be linked into the target program and be triggered to run from outside. Or triggered via external process injection, using CreateRemoteThread() on Windows, and on Linux using GDB debugger to attach to the process, to run command on dlopen() to load the hot loading library, and to call the library’s main routine.
What’s remaining is for the Zig compiler to figure out a list of modified functions since the last build and save it to a file for the hot loading library to use. I have no insight in this area.
Cheers!
3 Likes
If std.Build had an addCopyFile variation that could take a LazyPath and keep the old file name but change the rest of its path.
Getting rid of the abominable .{} syntax, for example replacing it with [].
1 Like
nah just put u16 or u32 packed structs in an extern struct.
Thanks for the reply! I’m under the impression that all the incremental and lazy compilation work that’s been done recently goes in the direction of also enabling hot patching since it seems quite similar in nature.
There are some libraries that try to solve it by compiling a DLL and then swapping the old for the new (popularized by Casey Muratori in Handmade Hero) and I’ve played around a bit with those and tried doing similar things myself. The thing that always seems difficult is keeping static data around without clearing it and making sure every pointer still points at the correct thing. Hot patching (which it sounds like you’re describing as well) should solve that sort of issue I hope.
It does sound like a quite difficult thing and therefore it’s something I’d like the language to solve for me without me having to always second guess whether it’s my program code or the hot reloading code that’s buggy. It’s a really important feature for game development which would be my primary use case for Zig.
Could you show example for the “Fragment Offset” specifically? Maybe I am misunderstanding something about Zig, but I think that particular thing can’t be represented as a single u13, no matter how you shuffle packed&extern? You need 5 least significant bits of a byte, and the next byte, and that conflicts with packed structs always going form LSB to MSB.
So the choices are to split it into lo & hi parts, or to write accessor functions, potentially generated via comptime DSL.
correct, there is no way to reinterpret memory as big-endian on a little-endian machine without a @byteSwap.
(packed to be renamed to bitpack in future release fyi)
const native_endian = @import("builtin").target.cpu.arch.endian();
const IPv4Header = packed struct (u16) {
fragment_offset: u13,
more_fragments: bool,
dont_fragment: bool,
reserved: u1 = 0,
fn deserialize(raw: [2]u8) IPv4Header {
return switch (native_endian) {
.big => @bitCast(raw),
.little => @bitCast(@byteSwap(@as(u16, @bitCast(raw)))),
};
}
};
3 Likes
The global static data is a problem. The newly reloaded functions will reference the global static data in the data segment of the new module, not the old ones. The workaround would be to make all the hot-reloadable functions stateless. Don’t reference global static data. Only work on data passed in as parameters. For hot reloading data, it’s better to reload them from data files.
Thinking about it more, I think all the pieces are in place. No need to change the Zig compiler. For the last piece of the puzzle, the changed functions can be detected from the debug info in PDB/DWARF. The debug info has the starting address and ending address of functions. The starting address and the length can be used to read the binary code bytes of the function body in the executable binary. Comparing the bytes of the new function against the bytes of the same function in the old executable can tell if it has been changed. Compare their lengths for a quicker check.
Here’s a quick back of the envelope design of a hot reloading tool:
- Only work on dynamic libraries. Confine often changed code in libraries.
- Save the baseline executable and PDB/DWARF files after a build. Start the baseline program.
- After a re-compilation, find the list of changed functions as above. Save the functions. Save the new relative addresses and the old baseline addresses of the functions in a file.
- Trigger hot-reload, either by the host program calling a function in the linked in hot-reload library (user might need to press a key), or by external code injection.
- Load new code. Dynamically load the new .dll/.so, and load the list of changed functions with their addresses from the file.
- Address resolution. Get the base address of the loaded baseline module. Get the base address of the loaded new module. Compute the absolute addresses of the baseline functions and the new functions.
- Patching. Modify the assembly near the beginning of a baseline function to jump to the new function’s address. Skip any stack frame prologue if needed. Caller calls the baseline function, which sets up the stack frame and jumps to the new function’s address beyond its stack frame prologue. New function should use the baseline’s stack frame. It’s better to just use an off-the-shelf library to do patching, not to mention needing to deal with stopping and resuming signal/interrupt/thread.
I think that covers all the bases. Make me want to start a new prototype. 
I’m already way behind on others. 
The workaround would be to make all the hot-reloadable functions stateless. Don’t reference global static data. Only work on data passed in as parameters. For hot reloading data, it’s better to reload them from data files.
That wouldn’t work for me unfortunately but thanks for taking the time to describe your ideas.