Modules are isomorphic to generic functions

One place in TigerBeetle where we get a lot of friction is that almost everything is a generic function, parametrized by Storage, IO, MessageBus and other types we inject in our simulator:

// src/vsr/replica.zig
pub fn ReplicaType(
    comptime StateMachine: type,
    comptime MessageBus: type,
    comptime Storage: type,
    comptime Time: type,
    comptime AOF: type,
) type {

This is the right kind friction in general, as generic types are costly; they should be minimized and concrete types should be preferred instead. At the same time, in our context I don’t think we can do that, as we pretty fundamentally need different real&fake implementations, and we don’t want dynamic dispatch in the real case, because our testing code should not affect production code (there are also performance considerations, but they are secondary here).

One thing I realized though is that we can remove generic parameters, by making them module parameters. If we make src/vsr/replica.zig and other things modules, than we we can do:

const StateMachine = @import("StateMachine");
const MessageBus = @import("MessageBus");
const Storage = @import("Storage");
const Time = @import("Time");
const AOF = @import("AOF");

I am wondering how horrible would it be to make this pattern first-class, by allowing a file to “import” its parameters, turning it into a function?

// src/vsr/replica.zig
const StateMachine,
const MessageBus,
const Storage,
const Time,
const AOF = @import();

// main.zig
const Replica = @import("./src/vsr.replica.zig")(
    StateMachine, 
    MessageBus,
    Storage,
    Time,
    AOF,
);

That is, a file with @import() is a type-returning function with comptime parameters.

3 Likes

Took me a while to understand it (still not sure if I did)… basically instead of @import being an implicit struct:

struct {
    [file content]
};

…it would wrap the file content in a function like this instead?

fn (T: type) type {
    return struct {    
        [file content]
    };
}

Did I get that right? On one hand, I actually missed this functionality too… mainly because the ‘type-constructor’ function causes the entire file content to be shifted one tab to the right which causes an awful lot of whitespace on the left side (e.g. chipz/src/chips/ay3891.zig at efb0176955d284cbbeac80025b56817dd566ed36 · floooh/chipz · GitHub)

OTH, I don’t like that this violates the simple ‘rule’ that an @import() always returns a struct.

Would the importer or the module decide whether the file content is wrapped in a struct or a type function (maybe @importType? What about @This()?

PS: also looking at a pattern like this where the type-constructor args are not a list of types, but an options arg instead (which then may contain types or regular constants: chipz/src/chips/ay3891.zig at efb0176955d284cbbeac80025b56817dd566ed36 · floooh/chipz · GitHub … how would that work with your idea?

…also what if the constructed type isn’t a struct but something else? I think it would either be too restrictive or would require additional syntax… hmm.

Yeah!

Would the importer or the module decide whether the file content is wrapped in a struct or a type function (maybe @importType ? What about @This() ?

The module. The rule is “if a file contains @import() syntactically, it is wrapped into a function”.

… how would that work with your idea?

Ah, snap, I indeed don’t seem to have a nice syntactic place to specify the type of comptime parameter, it has to be not only comptime, but actually anytype. I guess we can cook something up with RLS:

const config: ConfigType = @import();

but that’s stretching the idea quite a bit, and I don’t love it, thanks!

1 Like

…my main peeve is really the extra nesting level across almost the entire file because of the all-encompassing type-constructor function. I haven’t found a good solution for that yet, in some languages I use 2-space-wide tabs for that problem, but that makes regular if/else code too crowded.

…compared to the TypeName.zig convention it’s even two extra nesting levels…

@ThisScope(struct);

@ThisScope(fn (options: Options) type);

@ThisScope(enum)

…something something (e.g. hint the importer how the module content should be wrapped…

…but hmm… don’t know how I feel about this :smiley:

You can get most of this same behavior even without introducing new concepts.

Have your types be parametrized in a non-generic way with type parameters that come from other Zig modules, and then use the Zig build system to implement “generics” when composing an output executable (or a test one, or whatever artifact you’re generating).

In concrete (but a bit pseudozig) terms:

// Replica.zig
const StateMachine = @import("StateMachine");
const Storage = @import("Storage");

state: StateMachine,
storage: Storage, 
// ...
// build.zig

const exe = b.addExecutable(.{ 
   .name = "tigerbeetle" 
   // ...
});

exe.root_module.addImport("StateMachine", StateMachine);
exe.root_module.addImport("Storage", RealStorage);

const tests = b.addTest(...);
tests.root_module.addImport("StateMachine", StateMachine);
tests.root_module.addImport("Storage", TestStorage);

I’d be curious to know if you guys considered this approach at all and, if so, what issue you encountered with it that made you avoid it.

9 Likes

Would this switching between module implementations be something for build.zig?

I am not sure it is possible to define two different modules with the same name and then let the production executable depend on the real module and tests depend on the fake module.

Edit: Ah @kristoff was quicker with a way more elaborate answer :grin:

That (build.zig import massage), and a bit of if/else in the right hand side of the @import is what I use to parametrize my code across disparate backends (web, desktop).

I’m not happy about it yet. Main pet peeve is the extra level of indirection in practice (i.e., what I have to type, not lookups at run-time or something like that) because we can’t have a comptime if whose branch contents bubble up one level (a declaration in a comptime if has only the active branch as its scope, so you cannot import-resolve symbols from within the imported thing when you believe that usingnamespace will go). The extra state. or storage. or StateMachine. or Storage. that you’ll have to sprinkle gets old over time (but I don’t have a solution) but gets the job done.

Another challenge with the build.zig providing imports to the code is that nothing checks that the other thing that one might use also conforms to the shape you imply. Here comptime duck typing is quite loose again. You gotta manually make sure to always build all the options. This is a theme in zig though.

If so @import statement should be placed in obvious place in a file (beginning probably) and throw compile error if its not there.

Other way would be to make file always a function and just pass nothing if its’t not generic. But importing would be ugly

@import("something.zig")();