Modules are isomorphic to generic functions

One place in TigerBeetle where we get a lot of friction is that almost everything is a generic function, parametrized by Storage, IO, MessageBus and other types we inject in our simulator:

// src/vsr/replica.zig
pub fn ReplicaType(
    comptime StateMachine: type,
    comptime MessageBus: type,
    comptime Storage: type,
    comptime Time: type,
    comptime AOF: type,
) type {

This is the right kind friction in general, as generic types are costly; they should be minimized and concrete types should be preferred instead. At the same time, in our context I don’t think we can do that, as we pretty fundamentally need different real&fake implementations, and we don’t want dynamic dispatch in the real case, because our testing code should not affect production code (there are also performance considerations, but they are secondary here).

One thing I realized though is that we can remove generic parameters, by making them module parameters. If we make src/vsr/replica.zig and other things modules, than we we can do:

const StateMachine = @import("StateMachine");
const MessageBus = @import("MessageBus");
const Storage = @import("Storage");
const Time = @import("Time");
const AOF = @import("AOF");

I am wondering how horrible would it be to make this pattern first-class, by allowing a file to “import” its parameters, turning it into a function?

// src/vsr/replica.zig
const StateMachine,
const MessageBus,
const Storage,
const Time,
const AOF = @import();

// main.zig
const Replica = @import("./src/vsr.replica.zig")(
    StateMachine, 
    MessageBus,
    Storage,
    Time,
    AOF,
);

That is, a file with @import() is a type-returning function with comptime parameters.

3 Likes

Took me a while to understand it (still not sure if I did)… basically instead of @import being an implicit struct:

struct {
    [file content]
};

…it would wrap the file content in a function like this instead?

fn (T: type) type {
    return struct {    
        [file content]
    };
}

Did I get that right? On one hand, I actually missed this functionality too… mainly because the ‘type-constructor’ function causes the entire file content to be shifted one tab to the right which causes an awful lot of whitespace on the left side (e.g. chipz/src/chips/ay3891.zig at efb0176955d284cbbeac80025b56817dd566ed36 · floooh/chipz · GitHub)

OTH, I don’t like that this violates the simple ‘rule’ that an @import() always returns a struct.

Would the importer or the module decide whether the file content is wrapped in a struct or a type function (maybe @importType? What about @This()?

PS: also looking at a pattern like this where the type-constructor args are not a list of types, but an options arg instead (which then may contain types or regular constants: chipz/src/chips/ay3891.zig at efb0176955d284cbbeac80025b56817dd566ed36 · floooh/chipz · GitHub … how would that work with your idea?

…also what if the constructed type isn’t a struct but something else? I think it would either be too restrictive or would require additional syntax… hmm.

Yeah!

Would the importer or the module decide whether the file content is wrapped in a struct or a type function (maybe @importType ? What about @This() ?

The module. The rule is “if a file contains @import() syntactically, it is wrapped into a function”.

… how would that work with your idea?

Ah, snap, I indeed don’t seem to have a nice syntactic place to specify the type of comptime parameter, it has to be not only comptime, but actually anytype. I guess we can cook something up with RLS:

const config: ConfigType = @import();

but that’s stretching the idea quite a bit, and I don’t love it, thanks!

1 Like

…my main peeve is really the extra nesting level across almost the entire file because of the all-encompassing type-constructor function. I haven’t found a good solution for that yet, in some languages I use 2-space-wide tabs for that problem, but that makes regular if/else code too crowded.

…compared to the TypeName.zig convention it’s even two extra nesting levels…

@ThisScope(struct);

@ThisScope(fn (options: Options) type);

@ThisScope(enum)

…something something (e.g. hint the importer how the module content should be wrapped…

…but hmm… don’t know how I feel about this :smiley:

You can get most of this same behavior even without introducing new concepts.

Have your types be parametrized in a non-generic way with type parameters that come from other Zig modules, and then use the Zig build system to implement “generics” when composing an output executable (or a test one, or whatever artifact you’re generating).

In concrete (but a bit pseudozig) terms:

// Replica.zig
const StateMachine = @import("StateMachine");
const Storage = @import("Storage");

state: StateMachine,
storage: Storage, 
// ...
// build.zig

const exe = b.addExecutable(.{ 
   .name = "tigerbeetle" 
   // ...
});

exe.root_module.addImport("StateMachine", StateMachine);
exe.root_module.addImport("Storage", RealStorage);

const tests = b.addTest(...);
tests.root_module.addImport("StateMachine", StateMachine);
tests.root_module.addImport("Storage", TestStorage);

I’d be curious to know if you guys considered this approach at all and, if so, what issue you encountered with it that made you avoid it.

12 Likes

Would this switching between module implementations be something for build.zig?

I am not sure it is possible to define two different modules with the same name and then let the production executable depend on the real module and tests depend on the fake module.

Edit: Ah @kristoff was quicker with a way more elaborate answer :grin:

That (build.zig import massage), and a bit of if/else in the right hand side of the @import is what I use to parametrize my code across disparate backends (web, desktop).

I’m not happy about it yet. Main pet peeve is the extra level of indirection in practice (i.e., what I have to type, not lookups at run-time or something like that) because we can’t have a comptime if whose branch contents bubble up one level (a declaration in a comptime if has only the active branch as its scope, so you cannot import-resolve symbols from within the imported thing when you believe that usingnamespace will go). The extra state. or storage. or StateMachine. or Storage. that you’ll have to sprinkle gets old over time (but I don’t have a solution) but gets the job done.

Another challenge with the build.zig providing imports to the code is that nothing checks that the other thing that one might use also conforms to the shape you imply. Here comptime duck typing is quite loose again. You gotta manually make sure to always build all the options. This is a theme in zig though.

If so @import statement should be placed in obvious place in a file (beginning probably) and throw compile error if its not there.

Other way would be to make file always a function and just pass nothing if its’t not generic. But importing would be ugly

@import("something.zig")();

Apologies if necroing is not allowed, but I wanted to reply to @kristoff’s suggestion regarding this testing strategy and his request for feedback (also this topic is pretty specific so it seemed better to respond here rather than making a new topic).

I’ve been playing around over the past few years with different approaches regarding allowing easy separation/abstraction and testing between real code and testing code through having the usual “Real Implementation”, “Fake Implementation” and using Dependency Injection for my structures. I’ve tried most if not all of the ones that are available including:

  • Tagged Unions
  • Fat Pointer / VTable (Dynamic Dispatch)
  • Compile Time “Generics” (Static Dispatch) via anytype (Monomorphization)
  • Using conditional compilation to swap the implementation between Real/Fake:
    • const Communicator = if (!@import(“builtin”).is_test) @import(“communicator.zig”) else @import(“fakes/communicator.zig”);
  • And @kristoff’s Build System Module Substitution approach

I ultimately ended up converting my applications to using the builtin.is_test approach since it allowed the most flexibility, while retaining strong type safety (you need to maintain the function signatures the same and need to run your app in both real and test modes to make sure everything works, but you should already be doing this anyways), and allows the build system scripts to stay small, and still allows the writer to stay within the code flows itself rather than having to jump into the build system to understand how things are connected.

I really wanted the Module system approach to work since it made sense to me and it’s ultimately what I’m doing with builtin.is_test, while at the same time it allows me to not have to do the conditional compilation checks within every file that requires the substitution. However, the issues I’ve encountered were (and it may have been me not understanding things completely and may have needed to do additional re-wiring, but I did get the application to run in the real code paths):

  • I started getting errors that some modules or files were already imported between several files. This was weird because there was one fine (snapshot.zig) that was being used in a few places correctly, and was being used in one place simply to include the type definition of the Snapshot object itself, removing the import allowed the error to go away, but a new error appeared mentioning that I was using an undeclared type. So I wasn’t able to progress after that happened.
  • Another issue is that using the build system to do these substitutions actually makes the code harder to read because when you see something like const Communicator = @import(“Communicator”), the user wouldn’t really know if it’s the Real or Fake implementation without digging around in the build system to see what it evaluates to. This is of course a pro and a con and if the user is already familiar with the code base it’s an answer that once found, does make sense.
  • I may be forgetting another reason haha.

The new final design looks as follows and I’m pretty happy with it overall given the pros and cons of all of the various “interface/generics” strategies listed above:

  • The build script remains small (Pretty much the default zig init, although I’ll admit that I haven’t been using zig build test for running tests but instead use zig test src/tests.zig so simplify this. So there could be wiring improvements. However it’s a slightly similar strategy to how sometimes you would see a test.zig file across different folders in the standard library. In my case I’m just using one file in the main src folder. Before this code was inside the main.zig but after my conditional substitution change, there were compiler errors in the main.zig during test mode since the test framework correctly was trying to use the Fake version in the main.zig codepaths, but main.zig logic is not designed for testability on purpose. It is just the orchestrator for the “real world initial execution”).
  • The real Communicator implementation is imported directly in the main.zig since that file is meant only for “real executions”. The unit tests can all implement tests just for the public API of the application, the private APIs are not tested since they will be automatically tested via the public API. This also increases program maintenance and reduces friction when updating the app or doing refactorings (less tests = less broken tests that need updating due to testing too much internal implementation details).
  • Clean and easy separation of the “Real” and “Fake” implementations. They both implement the same “interface”, and my Fake implementation includes seams and injection points for anything I’m interested in checking (in a simple way) (things like “return values for functions”, “was called”, and “capturing received values during program flow”).
  • Easy and highly visible conditional compilation swapping of the Real and Fake implementations that can’t easily be missed by the programmer.
  • And lastly a few of the nice benefits of this approach (that is shared with the compile time generic approach) is that:
    • Things are statically dispatched so it’s more performant.
    • No need to re-write the function signatures an additional time like the tagged union approach.
    • Still maintains an “open” approach where downstream consumers could easily implement a new file that contains the exact function names and signature in order to satisfy the interface. The only downside is that the conditional logic needs to be updated in any file that requires the substitution based on the new criteria.

Hopefully the above is helpful feedback for the community.

Jonathan

1 Like