How to design a generic argument to a function

Help
1

I am currently writing a zig implementation of age, and one of the thing it do is allowing for multiple type of Identity (i.e, Scrypt, X25519, ssh-keys, etc…) to be use as the encryption and decryption keys for a given file. Since its designed to be extensible I couldn’t use tagged unions because 3rd party Identity using the library wouldn’t be able to use the provided encrypt/decrypt function.

In my current implementation, the encrypt/decrypt function have the following signature (simplifies compare to the actual implementation)

encrypt(plain_text: []const u8, recipients: []const AnyIdentity) []const u8
decrypt(encrypted_text: []const u8, identities: []const AnyIdentity) []const u8

all specific Identity type need to be convert into AnyIdentity interface (which i learn about from here), however because the interface relies on a *const anyopaque the specific Identity need to be on the same scope as when the Any interface is use.

So something like this wouldn’t work

var identities = ArrayList(AnyIdentity).init(alloc);
for (secret_keys) |key| {    
    if (key.prefix == "x25519") {
        // X25519Identity.parse(key) return a stack allocated memory
        const x25519 = X25519Identity.parse(key);
        // .any() return AnyIdentity with a context pointer pointed to x25519
        identities.append(x25519.any());
    }
    // else if ... for other types of supported keys
}
// x25519 go out of scope and AnyIdentity's context now pointed to invalid memory
// and the encryption would produce a file that can't be decrypt
encrypt("hello world", identities.items);

I have to do it like this

var x25519_identities = ArrayList(X25519Identity).init(alloc);
// more ArrayList(Identities).init() for supported keys
for (secret_keys) |key| {
    if (key.prefix == "x25519"){
        x25519_identities.append(X25519Identity.parse(key));
    }
    // else if ... for other types of supported keys
}
var any_identities = ArrayList(AnyIdentity).init(alloc);
for (x25519_identities) |identity| {
    identities.append(identity.any());
}
// repeat the block above for any extra specific identities (i.e scrypt, ssh, etc...)
encrypt("hello world", any_identities.items);

As you can see it is quite unwieldy.

I have a few idea:

  1. Use anytype: if the provided encrypt/decrypt function take in anytype instead of []const AnyIdentity then i could do something like this encrypt(encrypted_text, &.{x25519_identity, scrypt_identity, etc...}) but i can’t do ArrayList(anytype) to use a for loop to parse a list of inputs. anytype can’t be use since its compile time only.
  2. Use an allocator to duplicate the identity on the heap and free it later.

I don’t really want to do number 2 since it feel dirty (entirely out of my preconception that heap allocated memory should be avoids), how should i approach this?

1 Like
2

Hi, as I understand, we can do it like this:

const std = @import("std");

pub fn Identity(comptime T: type) type {
    return struct {
        id: T,

        pub fn init(id: T) @This() {
            return .{ .id = id };
        }
    };
}

pub fn encrypt(comptime T: type, identity: T, data: []const u8) ![]u8 {
    var encrypted = std.ArrayList(u8).init(std.heap.page_allocator);
    defer encrypted.deinit();

    try encrypted.appendSlice(identity.id);
    try encrypted.appendSlice(data);

    return encrypted.toOwnedSlice();
}

pub fn main() !void {
    const x25519_identity = Identity([]const u8).init("X25519-Key");
    const scrypt_identity = Identity([]const u8).init("Scrypt-Key");

    const data = "Hello, Zig!";

    const encrypted_x25519 = try encrypt(@TypeOf(x25519_identity), x25519_identity, data);
    defer std.heap.page_allocator.free(encrypted_x25519);

    const encrypted_scrypt = try encrypt(@TypeOf(scrypt_identity), scrypt_identity, data);
    defer std.heap.page_allocator.free(encrypted_scrypt);

    std.debug.print("Encrypted with X25519: {s}\n", .{encrypted_x25519});
    std.debug.print("Encrypted with Scrypt: {s}\n", .{encrypted_scrypt});
}

Note: I am not an expert in Zig yet, I am just trying to help, so you have to be sure and wait for the opinions of the experts.

3

Thanks for the reply, however compile time type wouldn’t be able to work here, since the type of identity use to encrypt/decrypt is something that’s given at runtime, I’ll edit the question to give more context.

While typing the sentence above it came to me that my 1st idea won’t work, because anytype is compile time value only.

1 Like
4

hmmm, what about it :

// Using anytype to allow passing any type at runtime
pub fn encrypt(identity: anytype, data: []const u8) ![]u8

and

// Passing the identity directly without needing to specify its type at compile time
const encrypted_x25519 = try encrypt(x25519_identity, data);

Edit:

I’m using anytype in a dynamic string type that works at runtime without problems in almost the same way.

Sorry if I don’t understand well, but I’m trying to learn here too xD

1 Like
5

this post have a good explanation of anytype

the problem with anytype is the compiler need to knows what type its suppose to be at compile time, lets say i have 2 identities type ScryptIdentity and X25519Identity which one to use is determined at run time, with anytype i have to call encrypt 2 times.

encrypt(x25519_identity, data);
encrypt(scrypt_identity, data);

i can’t do something like this

const identity: anytype = parseIdentity(key); // this doesnt compile
encrypt(identity, data);

I can’t call the encrypt function multiple time for each identity type because the age encryption spec requires the final encrypted file to have all identities used be in the header, calling encrypt multiples time would result in multiple files with a single identity in the header instead.

I suppose splitting the encrypt function into multiple stages, which is something i already did for the header and payload stage, splitting the header part further so there is an addIdentity stage that can accept anytype can work, thank you for that idea. I’ll wait to see if others have any other idea before i implement it.

1 Like
6

Oh, I see now, thank you for providing this great information and sorry for my lack of experience xD

1 Like
7

Looking at the source code, I don’t really understand why you need interfaces here. It seems the only useful function AnyIdentity implements is unwrap which returns a array. Couldn’t you pass these arrays directly? Or have some sort of any member on the structs that contains the generic context and is trivially copyable thus avoids the lifetime problem.

nevermind, I just realized the unwrap function takes in input that is needed to compute the result.

I’d probably change your api so you have encryptor/decryptor that works on streams, so you can push identities and then finalize the stream at the end and get the final result.

8

Do you mean instead of using AnyReader/AnyWriter, i should use StreamSource to keep everything in memory and only push it to the final destination, either a file or stdout, after the encrypt/decrypt process is done?

9

I mean instead having oneshot encrypt/decrypt functions. Have a stateful encryptor and decryptor. Sort of like how the stateful std.compress apis work.

1 Like
10

Thanks everyone for the suggestions

I’ve redesign it so encrypt/decrypt works in 4 parts

init: just return the struct
addIdentity: which take in an anytype and extract whatever is needed from an identity
finializeIdentities: which construct or destruct the header needed for encrypt/decrypt operations
And finally just encrypt or decrypt the data itself

1 Like