Unsure how to pass slices (or pointers in general) at comptime

Help
1

Hi all, I’m trying to mock up a quick wrapper around bufPrint to be able to turn a formatted string into unicode codepoints.

Here is the code that intends to do that:

pub fn formatAsCodepoints(comptime out_buffer: []Codepoint, comptime format_string: []const u8, arguments: anytype) anyerror![]Codepoint {
    var bytes_buffer:  [out_buffer.len*4]u8 = undefined;
    var printed_bytes: [                ]u8 = try bufPrint(&bytes_buffer, format_string, arguments);
    
    var oidx: usize = 0;
    while (printed_bytes.len > 0) {
        if (printed_bytes[0] & 0b10000000 == 0b00000000) {

            out_buffer[oidx] = .{ .word = @as(u32, printed_bytes[0]) };
            printed_bytes    = printed_bytes[1..];

        } else if (printed_bytes[0] & 0b11100000 == 0b11000000) {

            if (printed_bytes.len < 2) return error.UndefindeError;

            out_buffer[oidx] = .{ .word = @as(u32, try utf8Decode2(printed_bytes[0..2].*)) };
            printed_bytes    = printed_bytes[2..];

        } else if (printed_bytes[0] & 0b11110000 == 0b11100000) {

            if (printed_bytes.len < 3) return error.UndefindeError;

            out_buffer[oidx] = .{ .word = @as(u32, try utf8Decode3(printed_bytes[0..3].*)) };
            printed_bytes    = printed_bytes[3..];

        } else {

            if (printed_bytes.len < 4) return error.UndefindeError;

            out_buffer[oidx] = .{ .word = @as(u32, try utf8Decode4(printed_bytes[0..4].*)) };
            printed_bytes    = printed_bytes[4..];

        }
        oidx += 1;
    }
    return out_buffer[0..oidx];
}

I want to use the out_buffer to determine the size of bytes_buffer for bufPrint so I set it to comptime in the function parameters, but at the call site:

    var   codepoint_buffer: [1024]Codepoint = undefined;
    const formatted:        [    ]Codepoint = try tc.formatAsCodepoints(&codepoint_buffer, "hello there!", .{});
    print("{any}\n", .{formatted});

I am getting the error:

main.zig:49:73: error: unable to resolve comptime value
    const formatted:        [    ]Codepoint = try tc.formatAsCodepoints(&codepoint_buffer, "hello there!", .{});
                                                                        ^~~~~~~~~~~~~~~~~
main.zig:49:73: note: argument to comptime parameter must be comptime-known
textcrops.zig:281:27: note: parameter declared comptime here
pub fn formatAsCodepoints(comptime out_buffer: []Codepoint, comptime format_string: []const u8, arguments: anytype) anyerror![]Codepoint {

I figured doing things the way that I did would just work as you seem to be able to just do this with string literals, but is there something important about those technically being *const [:0]u8 that I haven’t considered?

I created this modified version that works:

pub fn formatAsCodepoints(out_buffer_pointer: anytype, comptime format_string: []const u8, arguments: anytype) anyerror![]Codepoint {
    if (!(
            @typeInfo(@TypeOf(out_buffer_pointer))                          == .pointer
        and @typeInfo(@typeInfo(@TypeOf(out_buffer_pointer)).pointer.child) == .array
    )) @compileError("expected pointer to array");

    const out_buffer_length:                     comptime_int = @typeInfo(@typeInfo(@TypeOf(out_buffer_pointer)).pointer.child).array.len;
    const out_buffer:        *[out_buffer_length]Codepoint    = @constCast(out_buffer_pointer);

    var bytes_buffer:  [out_buffer_length*4]u8 = undefined;
    var printed_bytes: [                   ]u8 = try bufPrint(&bytes_buffer, format_string, arguments);

    var oidx: usize = 0;
    while (printed_bytes.len > 0) {

        if (printed_bytes[0] & 0b10000000 == 0b00000000) {

            out_buffer.*[oidx] = .{ .word = @as(u32, printed_bytes[0]) };
            printed_bytes      = printed_bytes[1..];

        } else if (printed_bytes[0] & 0b11100000 == 0b11000000) {

            if (printed_bytes.len < 2) return error.UndefindeError;

            out_buffer.*[oidx] = .{ .word = @as(u32, try utf8Decode2(printed_bytes[0..2].*)) };
            printed_bytes      = printed_bytes[2..];

        } else if (printed_bytes[0] & 0b11110000 == 0b11100000) {

            if (printed_bytes.len < 3) return error.UndefindeError;

            out_buffer.*[oidx] = .{ .word = @as(u32, try utf8Decode3(printed_bytes[0..3].*)) };
            printed_bytes      = printed_bytes[3..];

        } else {

            if (printed_bytes.len < 4) return error.UndefindeError;

            out_buffer.*[oidx] = .{ .word = @as(u32, try utf8Decode4(printed_bytes[0..4].*)) };
            printed_bytes      = printed_bytes[4..];

        }

        oidx += 1;
    }
    return out_buffer.*[0..oidx];
}

Is the second way just the way that you’ve got to do it? Or is there a ‘nicer’ way to pass comptime known slices?

Any information or advice is greatly appreciated!

2

Slices generally don’t have a comptime known length, while arrays must have a comptime known length. So when you try to do:

    var bytes_buffer:  [out_buffer.len*4]u8 = undefined;

You create an array with a size of out_buffer.len*4. Because this value has to be known at comptime this means that outbuffer.len has to be known at comptime which means that has to be an array. This means that when you change the parameter type to be anytype and you use it as you do you basically declare that function as

pub fn formatAsCodepoints(
    out_buffer_pointer: *[1024]Codepoint,
    comptime format_string: []const u8, arguments: anytype,
) anyerror![]Codepoint`

as you’ve also checked right below the declaration.

String literals are basically saved in you executable “as-is” and read-only. Because of this their size is known at comptime and they can be treated as arrays. They can though also be treated as slices because a slice is just a pointer and a length and both are known at compile time.
Other comptime known slices are basically handled the same. They “decay” to an array internally.

Passing comptime slices is done rarely I think. You mostly just pass a slice.

Maybe you should take a look the print function in std.Io.Writer and in particular the u format specifier:

/// - `u`: output integer as an UTF-8 sequence. Integer type must have 21 bits at max.

TL;DR: Arrays have a comptime known lenght.

I hope that helps somewhat.

2 Likes
3

Ye that makes sense. I guess I thought the compiler might be able to use the context to do some implicit work, but I guess that goes against the Zig ethos. String literals are probably the one place where it’s valid to have a bit of magic processing since they’re well-defined and constrained, and the language might be too cumbersome without it. Well in any case I can still do what I set out do, I just had to be a little more explicit about it

As for your suggestion at the end of your message, I’m actually trying to have the data in the form of unicode code points (the u21’s that it mentions). I want to do some transformation on the data that is independent of the input encoding

Anyway, thank you for your reply

All the best

4

Just to further your understanding a bit more: That’s not really special to string literals. All comptime known constants behave basically the same way(this is also the same for most compiled languages).

I think you’ve likely already done this but take a look into std.unicode. Maybe the Utf8View and Uft8Iterator could be used for the parsing.
So conceptually something like this(mostly from memory so no guarantees):

fn toCodepoints(buf: []const u8, gpa: mem.Allocator) ![]Codepoint {
    var codepoints: std.ArrayList(Codepoint) = .empty;
    errdefer codepoints.deinit(gpa);
    const view: std.unicode.Utf8View = .init(buf);
    var iter = view.iterator();
    while (iter.next()) |cp| {
        codepoints.append(gpa, .{.inner = cp});
    }
    return try codepoints.toOwnedSlice(gpa);
}

You could then also have an initCapacity or ensureTotalCapacity call below the errdefer to reduce the number of allocations done.

1 Like
5

not relevant to the question, but I have to point out you should not use anyerror, as that means every single error possible anywhere in the program is a valid value, so the compiler, and the caller by extension, won’t know the actual set of errors that are possible.

you probably what an inferred error set instead, that is done by omitting the error entirely e.g !void. Though I would recommend you create defined error sets where practical.

2 Likes
6

oh wow, I did not see that iterator in std. Thanks for the tip, I’ll definitely look into that

7

ye I know, but my programming style is to do error handling in so far as it helps me shape the happy path. Then later I get down to the nitty-gritty of error handling. So I’ll just mark fallible functions with anyerror, put a bunch of return error.UndefinedError’s where those errors could happen and then continue with the main logic. I then come back later to add the specific error sets, and where necessary, put alarms (what I calling error logging) with whatever relevant data. For reasons that only the creators know, I have very little mental bandwidth and can get myself turned around pretty quickly, so doing this helps me focus on one thing at a time. Thank you for information nontheless

edit: I should mention that I use anyerror instead of letting the compiler infer things for me because the inference process can become cyclic on recursive functions. I know this to be an issue with inference because you can fix it by specifying the error set or by using anyerror