Embed binary data at compile time

Hey :slight_smile: this is my first Post and also my first attempts at metaprogramming.

For my Project im working on, i needed some icons and found them in the form of SVGs.
Because i want this project to be small and minimal, and because i need the icons the whole time the Program is running, i want to embed these.
… so far so easy …
But DVUI (the UI lib im using) only talks TVG not SVG.
There is a function provided by DVUI dvui.svgToTvg(...) which works fine, but it wants to alloc those, that way my program would have the SVGs embedded and the TVGs on the heap, also … i would create the TVGs every time the Program is setting up.
I don’t want this.

I could just convert all of my assets by hand,… but i also dont like this. I want my assets to be just a folder of SVGs!

So… i managed to get the following working and would like to know if there is anything bad/dirty with it and most importantly: Is there an easier way? :slight_smile: (comptime blocks dont support Io in my findings, right?)
Also, are my generated TVGs really embedded this way?

My meta file:

const std = @import("std");
const dvui = @import("dvui");

pub fn main(init: std.process.Init) !void {
    const arena = init.arena.allocator();

    const args = try init.minimal.args.toSlice(arena);
    const cwd = std.Io.Dir.cwd();

    const assets = try cwd.openDir(init.io, "src/assets", .{ .iterate = true });
    defer assets.close(init.io);

    var output: std.ArrayList(u8) = try .initCapacity(arena, 100 * 1024);

    try output.print(
        arena,
        "pub const Tvgs = struct {s}\n",
        .{"{"},
    );

    var names: std.ArrayList([]const u8) = .empty;

    var asset_iter = try assets.walk(arena);
    while (try asset_iter.next(init.io)) |svg| {
        const svg_data = try assets.readFileAlloc(init.io, svg.path, arena, .unlimited);
        const tvg = try dvui.svgToTvg(arena, svg_data);
        var name_buf: [1024]u8 = undefined;
        _ = std.mem.replace(u8, svg.basename, "-", "_", &name_buf);
        const name = try arena.alloc(u8, name_buf[0 .. svg.basename.len - 4].len);
        std.mem.copyForwards(u8, name, name_buf[0 .. svg.basename.len - 4]);
        try output.print(
            arena,
            "const {s}_const: [{d}]u8 = .{s} ",
            .{ name, tvg.len, "{" },
        );
        try names.append(arena, name);

        for (tvg) |b| {
            try output.print(
                arena,
                "{d}, ",
                .{b},
            );
        }

        try output.print(
            arena,
            "{s};\n",
            .{"}"},
        );
    }

    for (names.items) |name| {
        try output.print(
            arena,
            "{s}: []const u8 = &Tvgs.{s}_const,\n",
            .{ name, name },
        );
    }

    try output.print(
        arena,
        "{s};\n",
        .{"}"},
    );

    var output_file = try cwd.createFile(init.io, args[1], .{});
    defer output_file.close(init.io);

    try output_file.writeStreamingAll(init.io, output.items);

    return std.process.cleanExit(init.io);
}

it gets embedded like this example here.

You escape braces in format strings by doubling them.

What I dislike about it is that it generates one big source file that encodes binary data as text (which then gets converted back into binary by the compiler), if any of the input files are changed you need to re-convert all files not just the ones that changed (depends on your use-case whether that is a problem).


But I am also unsure whether there is an easy build-cache friendly alternative for handling a folder of files that you want to transform and then embed, without re-doing unnecessary work.

If your build script walks the directory to enumerate the individual files, it could create a run-step for each individual file to convert it from svg to tvg, then you could use one addWriteFiles to collect all the transformed output files in one directory that should be cached by the build system and only re-run the individual conversions for input-svgs that changed. (haven’t tested this specific scenario)

That wouldn’t handle adding or removing files while using watch or incremental builds, but should allow editing existing files.

I think the resulting wf.getDirectory() then could be passed on to Zig Asset Bundler

There is also this older related topic:


Overall I would say, I don’t think your solution is ideal, or should be the go to solution long term, but if it works for you, it might be the pragmatic solution for now, until somebody shows how to handle this more simply / idiomatically.

3 Likes

AHA! I tried the naive \{ and the compiler told my kindly that its wrong, i wondered if there is a way to properly escape them :slight_smile:

I guess i should check the timings of the compilation, for now its not a problem i think. Also source SVGs will change rarely. But indeed propper incremental compilation with the assets sounds like a nice idea.

While this sounds fun (compression!), i’d rather have a solution without a new dependency. Also this will then allocate the data, which i belive i don’t need and therefore don’t want.

This reads a bit like my code could be simplified a lot :smiley:

I would say my solution is far from ideal, but im proud of my trash because it kind of does the thing i want it to for now!

Thank you very much for taking the time to answer :slight_smile:

2 Likes

Whats wrong with:

const svgs: []const []const u8 = &.{
  @embedFile("1.svg"),
  @embedFile("2.svg"),
  ...
};
  • They prefer to keep the assets as SVG on-disk
  • Internally the library being used expects TVG
  • This means embedding SVG into RO memory of the binary, just to be converted at runtime to TVG in dynamic heap memory (duplication)
  • Ideally they wish to use the build system to convert/embed SVG to TVG so that they can have best of both world, but want to do it in an optimal way.

I think that summarizes the situation succinctly.

4 Likes

That is absolutely correct. :+1:

What about a build step in build.zig that converts the SVG to TVG? That would move it to compile-time only, separate the conversion from the code, the runtime wouldn’t include the SVGs, and wouldn’t need to convert during runtime, and you could use @embedFile() as @LucasSantos91 suggested.

(Not trying to discount any of the effort you put into this.)

3 Likes

This is a possible and cleaner Solution, but not for me.
in my version i can just use it like this:

const Tvgs = @import("tvgs.zig").Tvgs;

...

dvui.functionThatUsesIcon(bla, Tvgs.settings_icon_const);

The difference is my assets all get bundled and i don’t have to manually import them,
and i can refer to them via name.

Also, but this would be okay i guess, in my version there are no .tvg files on disk.

Edit:
In the future if i need a new Icon, basically i just want to:

  • download SVG
  • put it into asset dir
  • directly use it

Yeah tbh I would simply replace the ‘hex-dumping’ code-generation with writing binary data, e.g. replace the above code with basically (pseudocode):

  • svg_ata = readFile(“bla.svg”)
  • const tvg_data = try dvui.svgToTvg(arena, svg_data);
  • writeFile(“bla.tvg”, tvg_data)

…compile that code into a cmdline tool, run at build time, and then in the ‘actual’ executable embed the generated tvg data via @embed.

Basically how it is described here: Zig Build System ⚡ Zig Programming Language

3 Likes

Can you explain why it doesn’t work for you? I’m trying to understand.

If you want to use a new icon in your code, you will need to compile the code, right? The build system will check if there’s a corresponding tvg and if not, it will create it. You could still download an svg, put it in the asset directory, compile the code, and use it. Your code, as presented, will work as-is.

The only downside is that your code won’t be able to do it on the fly and will need a build for new assets. But from that point on, it’s smooth sailing. (Also, doing this sort of operation in runtime is not a good pattern, IMHO.)

Again, just trying to understand.

2 Likes

No worries, i’m the one trying to get help :slight_smile:

I think i will incorporate your solution but with an added bit. (Maybe that’s what you meant anyways)

  • Do the build step for each asset which converts svg to tvg (like in the topic @Sze mentioned)
  • handoff the filenames to another buildstep which generates a .zig file which @embed(filename) into a struct ? (is this possible?)
  • use it like i do now

how i understood your solution was:
everything is done at build for me, but i would still need to manually @embed(generated.tvg) for every icon. Which i would rather not, when i add for example 100 new ones…

Essentially i’m trying to be very lazy. :slight_smile:

Thank you all for taking your time, it is very possible i just don’t understand the solutions you offer because im still an Azubi (apprentice)

when i add for example 100 new ones…

Oh! I see what you mean here! You’re saying you’re using a very large amount you don’t want to write by hand.

But then, how do you use them? Don’t you need to still use their name when determining where you place them in DVUI? Or do you just place an array of TVGs and you don’t care what their names are? Maybe you explained it and I missed it.

In my GUI work, my code says “here put this one, over there put that one,” so no matter how many those are, I need to indicate which goes where.

1 Like

yes

That’s how i use them.

That why no matter how many i add i have one struct containing all of them and the LSP will kindly give me a list of every icon i added. That’s why i need the name of the file as field name. So i will just type the name (with the help of the LSP) just when they are really needed. :slight_smile:

Again i think i’m just being lazy :smiley:

const std = @import("std");

const files = blk: {
    const Files = struct {
        foo: []const u8,
        bar: []const u8,
    };
    var f: Files = undefined;
    for (@typeInfo(Files).@"struct".fields) |field| {
        @field(f, field.name) = @embedFile(field.name ++ ".txt");
    }
    break :blk f;
};

pub fn main() void {
    std.debug.print("{s}\n", .{files.foo});
    std.debug.print("{s}\n", .{files.bar});
}
Hi from Foo.
Hello there from bar!
2 Likes

Again i think i’m just being lazy :smiley:

That’s a perfectly fine usage for them, and getting the LSP benefits is great. That doesn’t contradict my idea. The point is to separate both when and where you’re converting the SVG to TVG. That’s all. The way you expose it to your app is fully up to you.

@ScottRedig’s proposal is the way to expose it.

I get the feeling that a bunch of answers have not read the whole topic, or spend no time to look at @schlyngel’s code.

I think from a pragmatic view point that code is already better than a bunch of the suggestions here, for example it allows to just drop the svg file into the folder and then refer to the generated tvg by it’s name, without manually re-declaring a field name identical to the image file.

And how do you do that without the code becoming more awkward and complex than what @schlyngel already has?
Or alternatively at least overall better even if it is more code (for example with better incremental support)?

Please make it concrete, instead of talking in general terms, if you know how to do it better than show concrete steps, or examples where you have done so.

Personally I think @schlyngel already has the best solution from a practical perspective, until the compiler implements the communication protocol (so that added/deleted files can trigger incremental re-generation properly) and some kind of support for applying a run step to all files in a directory matching a pattern. (Maybe there is a way to hack this together already, if you have done something like this please show it)

2 Likes

“Awkward and complex” is a judgment call, so no matter what I propose, you might fairly consider it awkward and complex. In this sense, @schlyngel is welcome to reject any proposals for the same argument. “I find it awkward and complex” is a legitimate response.

However, to me, subjectively, having the build system align the requirements of runtime is more elegant. You are welcome to see differently and we can all be friends. :slight_smile:

Respectfully, I don’t care for this. The argument of “put up or shut up” when someone asks for advice is unfair. You’re essentially saying, “if I ask you for advice, you are either concrete to my liking or you STFU.” That’s not nice, to say the least.

My proposal was concrete to me and I provided the general idea. I’m not required to be concrete enough to you to share my opinion and advice.

Beyond this, when we seek advice, we have to assume on best intentions and that the person answering us might not have the time to write up steps, code, etc. My approach could’ve been flawed, non-working, ridiculous on its face, but it was made with genuine desire to help another person and this response does not encourage further conversation, brainstorming, or cooperation.

1 Like

Nah, making stuff concrete helps with not just talking fluff, that ultimately doesn’t help anyone and creates more confusion.

It is fine to leave drive by general advice, but then mark it as such, instead of making it sound like you have the insider info of having dealt with the nuance, not being willing to invest the time to actually show how to do it.

How much time you invest is your thing, but don’t present answers in a way where it makes others think like there is an obvious simple answer, that they aren’t given any further hint about, that you can’t be bothered to share.

If someone opens a help topic they don’t want general platitudes.

He is already using the build system to transform the files at build time, generates a module which contains the data of them. The shown program runs as a runstep at build time to generate a module which then provides the data.

Sure we can be friends, I just don’t want to talk at this level of abstract communication, I don’t find it productive.

I understand you’re at the point of just doubling down on your behavior. Perhaps you might instead try to focus on the impact your behavior causes.

Telling me to keep quiet because I’m not concrete enough to your liking tells me and others not to interact further. I believe that’s a net loss for the community. If you’re happy with it, I have nothing further to add.

Well i tried to change some of the suggested things, but i didn’t manage to get it working. Nonetheless i thank you all again for taking your time trying to help me :slight_smile:

I just changed small things now like using {{ and i realized i don’t need the fields in the struct, only the pub const something ... inside the struct is actually enough.

If i’m not adding any new SVGs (or delete them) the compiler will cache this. So in a sense my trash is good enough for now :smiley:

If someone in the future figures out how to do this more cleanly and can explain it so that i can understand, i would be very happy!