Embed binary data at compile time

Help
,
1

Hey :slight_smile: this is my first Post and also my first attempts at metaprogramming.

For my Project im working on, i needed some icons and found them in the form of SVGs.
Because i want this project to be small and minimal, and because i need the icons the whole time the Program is running, i want to embed these.
… so far so easy …
But DVUI (the UI lib im using) only talks TVG not SVG.
There is a function provided by DVUI dvui.svgToTvg(...) which works fine, but it wants to alloc those, that way my program would have the SVGs embedded and the TVGs on the heap, also … i would create the TVGs every time the Program is setting up.
I don’t want this.

I could just convert all of my assets by hand,… but i also dont like this. I want my assets to be just a folder of SVGs!

So… i managed to get the following working and would like to know if there is anything bad/dirty with it and most importantly: Is there an easier way? :slight_smile: (comptime blocks dont support Io in my findings, right?)
Also, are my generated TVGs really embedded this way?

My meta file:

const std = @import("std");
const dvui = @import("dvui");

pub fn main(init: std.process.Init) !void {
    const arena = init.arena.allocator();

    const args = try init.minimal.args.toSlice(arena);
    const cwd = std.Io.Dir.cwd();

    const assets = try cwd.openDir(init.io, "src/assets", .{ .iterate = true });
    defer assets.close(init.io);

    var output: std.ArrayList(u8) = try .initCapacity(arena, 100 * 1024);

    try output.print(
        arena,
        "pub const Tvgs = struct {s}\n",
        .{"{"},
    );

    var names: std.ArrayList([]const u8) = .empty;

    var asset_iter = try assets.walk(arena);
    while (try asset_iter.next(init.io)) |svg| {
        const svg_data = try assets.readFileAlloc(init.io, svg.path, arena, .unlimited);
        const tvg = try dvui.svgToTvg(arena, svg_data);
        var name_buf: [1024]u8 = undefined;
        _ = std.mem.replace(u8, svg.basename, "-", "_", &name_buf);
        const name = try arena.alloc(u8, name_buf[0 .. svg.basename.len - 4].len);
        std.mem.copyForwards(u8, name, name_buf[0 .. svg.basename.len - 4]);
        try output.print(
            arena,
            "const {s}_const: [{d}]u8 = .{s} ",
            .{ name, tvg.len, "{" },
        );
        try names.append(arena, name);

        for (tvg) |b| {
            try output.print(
                arena,
                "{d}, ",
                .{b},
            );
        }

        try output.print(
            arena,
            "{s};\n",
            .{"}"},
        );
    }

    for (names.items) |name| {
        try output.print(
            arena,
            "{s}: []const u8 = &Tvgs.{s}_const,\n",
            .{ name, name },
        );
    }

    try output.print(
        arena,
        "{s};\n",
        .{"}"},
    );

    var output_file = try cwd.createFile(init.io, args[1], .{});
    defer output_file.close(init.io);

    try output_file.writeStreamingAll(init.io, output.items);

    return std.process.cleanExit(init.io);
}

it gets embedded like this example here.

2

You escape braces in format strings by doubling them.

What I dislike about it is that it generates one big source file that encodes binary data as text (which then gets converted back into binary by the compiler), if any of the input files are changed you need to re-convert all files not just the ones that changed (depends on your use-case whether that is a problem).

But I am also unsure whether there is an easy build-cache friendly alternative for handling a folder of files that you want to transform and then embed, without re-doing unnecessary work.

If your build script walks the directory to enumerate the individual files, it could create a run-step for each individual file to convert it from svg to tvg, then you could use one addWriteFiles to collect all the transformed output files in one directory that should be cached by the build system and only re-run the individual conversions for input-svgs that changed. (haven’t tested this specific scenario)

That wouldn’t handle adding or removing files while using watch or incremental builds, but should allow editing existing files.

I think the resulting wf.getDirectory() then could be passed on to Zig Asset Bundler

There is also this older related topic:

Overall I would say, I don’t think your solution is ideal, or should be the go to solution long term, but if it works for you, it might be the pragmatic solution for now, until somebody shows how to handle this more simply / idiomatically.

1 Like
3

AHA! I tried the naive \{ and the compiler told my kindly that its wrong, i wondered if there is a way to properly escape them :slight_smile:

I guess i should check the timings of the compilation, for now its not a problem i think. Also source SVGs will change rarely. But indeed propper incremental compilation with the assets sounds like a nice idea.

While this sounds fun (compression!), i’d rather have a solution without a new dependency. Also this will then allocate the data, which i belive i don’t need and therefore don’t want.

This reads a bit like my code could be simplified a lot :smiley:

I would say my solution is far from ideal, but im proud of my trash because it kind of does the thing i want it to for now!

Thank you very much for taking the time to answer :slight_smile:

1 Like