Unsure how to pass slices (or pointers in general) at comptime

Help
1

Hi all, I’m trying to mock up a quick wrapper around bufPrint to be able to turn a formatted string into unicode codepoints.

Here is the code that intends to do that:

pub fn formatAsCodepoints(comptime out_buffer: []Codepoint, comptime format_string: []const u8, arguments: anytype) anyerror![]Codepoint {
    var bytes_buffer:  [out_buffer.len*4]u8 = undefined;
    var printed_bytes: [                ]u8 = try bufPrint(&bytes_buffer, format_string, arguments);
    
    var oidx: usize = 0;
    while (printed_bytes.len > 0) {
        if (printed_bytes[0] & 0b10000000 == 0b00000000) {

            out_buffer[oidx] = .{ .word = @as(u32, printed_bytes[0]) };
            printed_bytes    = printed_bytes[1..];

        } else if (printed_bytes[0] & 0b11100000 == 0b11000000) {

            if (printed_bytes.len < 2) return error.UndefindeError;

            out_buffer[oidx] = .{ .word = @as(u32, try utf8Decode2(printed_bytes[0..2].*)) };
            printed_bytes    = printed_bytes[2..];

        } else if (printed_bytes[0] & 0b11110000 == 0b11100000) {

            if (printed_bytes.len < 3) return error.UndefindeError;

            out_buffer[oidx] = .{ .word = @as(u32, try utf8Decode3(printed_bytes[0..3].*)) };
            printed_bytes    = printed_bytes[3..];

        } else {

            if (printed_bytes.len < 4) return error.UndefindeError;

            out_buffer[oidx] = .{ .word = @as(u32, try utf8Decode4(printed_bytes[0..4].*)) };
            printed_bytes    = printed_bytes[4..];

        }
        oidx += 1;
    }
    return out_buffer[0..oidx];
}

I want to use the out_buffer to determine the size of bytes_buffer for bufPrint so I set it to comptime in the function parameters, but at the call site:

    var   codepoint_buffer: [1024]Codepoint = undefined;
    const formatted:        [    ]Codepoint = try tc.formatAsCodepoints(&codepoint_buffer, "hello there!", .{});
    print("{any}\n", .{formatted});

I am getting the error:

main.zig:49:73: error: unable to resolve comptime value
    const formatted:        [    ]Codepoint = try tc.formatAsCodepoints(&codepoint_buffer, "hello there!", .{});
                                                                        ^~~~~~~~~~~~~~~~~
main.zig:49:73: note: argument to comptime parameter must be comptime-known
textcrops.zig:281:27: note: parameter declared comptime here
pub fn formatAsCodepoints(comptime out_buffer: []Codepoint, comptime format_string: []const u8, arguments: anytype) anyerror![]Codepoint {

I figured doing things the way that I did would just work as you seem to be able to just do this with string literals, but is there something important about those technically being *const [:0]u8 that I haven’t considered?

I created this modified version that works:

pub fn formatAsCodepoints(out_buffer_pointer: anytype, comptime format_string: []const u8, arguments: anytype) anyerror![]Codepoint {
    if (!(
            @typeInfo(@TypeOf(out_buffer_pointer))                          == .pointer
        and @typeInfo(@typeInfo(@TypeOf(out_buffer_pointer)).pointer.child) == .array
    )) @compileError("expected pointer to array");

    const out_buffer_length:                     comptime_int = @typeInfo(@typeInfo(@TypeOf(out_buffer_pointer)).pointer.child).array.len;
    const out_buffer:        *[out_buffer_length]Codepoint    = @constCast(out_buffer_pointer);

    var bytes_buffer:  [out_buffer_length*4]u8 = undefined;
    var printed_bytes: [                   ]u8 = try bufPrint(&bytes_buffer, format_string, arguments);

    var oidx: usize = 0;
    while (printed_bytes.len > 0) {

        if (printed_bytes[0] & 0b10000000 == 0b00000000) {

            out_buffer.*[oidx] = .{ .word = @as(u32, printed_bytes[0]) };
            printed_bytes      = printed_bytes[1..];

        } else if (printed_bytes[0] & 0b11100000 == 0b11000000) {

            if (printed_bytes.len < 2) return error.UndefindeError;

            out_buffer.*[oidx] = .{ .word = @as(u32, try utf8Decode2(printed_bytes[0..2].*)) };
            printed_bytes      = printed_bytes[2..];

        } else if (printed_bytes[0] & 0b11110000 == 0b11100000) {

            if (printed_bytes.len < 3) return error.UndefindeError;

            out_buffer.*[oidx] = .{ .word = @as(u32, try utf8Decode3(printed_bytes[0..3].*)) };
            printed_bytes      = printed_bytes[3..];

        } else {

            if (printed_bytes.len < 4) return error.UndefindeError;

            out_buffer.*[oidx] = .{ .word = @as(u32, try utf8Decode4(printed_bytes[0..4].*)) };
            printed_bytes      = printed_bytes[4..];

        }

        oidx += 1;
    }
    return out_buffer.*[0..oidx];
}

Is the second way just the way that you’ve got to do it? Or is there a ‘nicer’ way to pass comptime known slices?

Any information or advice is greatly appreciated!

2

Slices generally don’t have a comptime known length, while arrays must have a comptime known length. So when you try to do:

    var bytes_buffer:  [out_buffer.len*4]u8 = undefined;

You create an array with a size of out_buffer.len*4. Because this value has to be known at comptime this means that outbuffer.len has to be known at comptime which means that has to be an array. This means that when you change the parameter type to be anytype and you use it as you do you basically declare that function as

pub fn formatAsCodepoints(
    out_buffer_pointer: *[1024]Codepoint,
    comptime format_string: []const u8, arguments: anytype,
) anyerror![]Codepoint`

as you’ve also checked right below the declaration.

String literals are basically saved in you executable “as-is” and read-only. Because of this their size is known at comptime and they can be treated as arrays. They can though also be treated as slices because a slice is just a pointer and a length and both are known at compile time.
Other comptime known slices are basically handled the same. They “decay” to an array internally.

Passing comptime slices is done rarely I think. You mostly just pass a slice.

Maybe you should take a look the print function in std.Io.Writer and in particular the u format specifier:

/// - `u`: output integer as an UTF-8 sequence. Integer type must have 21 bits at max.

TL;DR: Arrays have a comptime known lenght.

I hope that helps somewhat.

1 Like
3

Ye that makes sense. I guess I thought the compiler might be able to use the context to do some implicit work, but I guess that goes against the Zig ethos. String literals are probably the one place where it’s valid to have a bit of magic processing since they’re well-defined and constrained, and the language might be too cumbersome without it. Well in any case I can still do what I set out do, I just had to be a little more explicit about it

As for your suggestion at the end of your message, I’m actually trying to have the data in the form of unicode code points (the u21’s that it mentions). I want to do some transformation on the data that is independent of the input encoding

Anyway, thank you for your reply

All the best