Rendering text with harfbuzz/freetype

I’d like to create a small Flashcard program to teach myself Hindi. This means that I need to render text in the Devanagari script, like this "“मेरा नाम लार्स है”.

With some help from the Mach discord, I’ve set up mach-freetype and managed to render this:

Well, it does show glyphs from a Devanagari font. But the layout is quite broken. It would be a huge help if someone with harfbuzz experience could take a look at my code. In particular it’s not clear to me how to correctly position the glyphs. The advances and offsets should be quite straightforward but they don’t work as expected. Maybe there’s some interplay with setting a font size, but so far it’s not clear to me.

To render the harfbuzz/freetype bitmaps I’m using raylib. This is not a requirement, I could also use something else. For now it was just very easy to get up and running with minimal boilerplate:

const std = @import("std");
const freetype = @import("mach-freetype");
const harfbuzz = @import("mach-harfbuzz");
const font_assets = @import("font-assets");

const rl = @import("raylib");
const rg = @import("raygui");

pub fn main() !void {
    // raylib setup
    const screenWidth = 800;
    const screenHeight = 400;
    rl.initWindow(screenWidth, screenHeight, "devanagari text");
    defer rl.closeWindow();
    rl.setTargetFPS(60);

    // Normally I use the GPA allocator, but it seems incompatible with raylib.
    // See further below for a comment on this issue.
    // var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    // const allocator = gpa.allocator();
    // defer {
    //     _ = gpa.deinit();
    // }
    const allocator = std.heap.c_allocator;

    const text = "मेरा नाम लार्स है";
    //const text = "मेर";

    // set up harfbuzz buffer, feed it the Devanagari text
    var hb_buffer = harfbuzz.Buffer.init() orelse return;
    hb_buffer.setDirection(harfbuzz.Direction.ltr);
    hb_buffer.setScript(harfbuzz.Script.devanagari);
    hb_buffer.addUTF8(text, 0, text.len);

    // set up freetype face, convert to harfbuzz face, and shape the buffer
    const lib = try freetype.Library.init();
    defer lib.deinit();
    const ft_face = try lib.createFaceMemory(@embedFile("NotoSerifDevanagari_Condensed-Bold.ttf"), 0);

    // ToDo: it's not clear to me how to set the font size correctly
    try ft_face.setCharSize(100 * 50, 0, 50, 0);
    const hb_face = harfbuzz.Face.fromFreetypeFace(ft_face);
    const hb_font = harfbuzz.Font.init(hb_face);
    hb_font.shape(hb_buffer, null);

    // glyph infos and positions, that's what we need to render the text
    const glyphInfos = hb_buffer.getGlyphInfos();
    const glyphPositions = hb_buffer.getGlyphPositions() orelse return;

    while (!rl.windowShouldClose()) {

        // some raylib boilerplate
        // the blendMode is important:
        // harfbuzz and freetype ultimately give me bitmaps and positions
        // the positions overlap
        // if I'm drawing white text on a black background, then I can just use blend_additive to draw the glyphs
        // ultimately I would like to get black text on white background, but I think that will require a more complex setup
        rl.beginDrawing();
        defer rl.endDrawing();
        rl.clearBackground(rl.Color.black);
        rl.beginBlendMode(rl.BlendMode.blend_additive);
        defer rl.endBlendMode();

        // start position for the text
        var pos_x: i32 = 100;
        var pos_y: i32 = 100;

        for (glyphInfos, glyphPositions) |info, pos| {
            std.debug.print("codepoint: {d}\n", .{info.codepoint});
            std.debug.print("{any}\n", .{pos});

            // get a bitmap for the current glyph
            try ft_face.loadGlyph(info.codepoint, .{ .render = true });
            const glyph = ft_face.glyph();
            const bm = glyph.bitmap();
            std.debug.print("bitmap: {d}x{d}\n", .{ bm.width(), bm.rows() });

            // create an image from the bitmap data
            // this block is raylib specific
            var image: rl.Image = undefined;
            image.width = @intCast(bm.width());
            image.height = @intCast(bm.rows());
            image.mipmaps = 1;
            image.format = rl.PixelFormat.pixelformat_uncompressed_grayscale;

            // Do I need to create this copy?
            // using the bitmap data directly requires a @constCast(bm.buffer().?.ptr) which looks like it can't possibly be correct.
            const buffer = bm.buffer();
            if (buffer == null) {
                continue;
            }
            const bitmapBuffer = try allocator.dupe(u8, buffer.?);

            // quite strange: If I uncomment this line, I get a double free error
            // does raylib free some texture buffer itself?
            // I'm not using raylib unloadImage or unloadTexture
            // to me this looks like a big memory leak
            // but: If I don't use the c_allocator, I get a "freed pointer wasn't allocated" error
            // so I guess raylib is trying to free this buffer and if it was allocated with a different allocator, then it fails to do so
            //defer allocator.free(bitmapBuffer);
            image.data = bitmapBuffer.ptr;

            // to draw, we need to convert the image into a texture
            const texture: rl.Texture2D = rl.loadTextureFromImage(image);

            // ToDo: the values for advance are much too high
            // if I apply them, the textures are drawn offscreen
            // pos_x += pos.x_advance;
            // pos_y += pos.y_advance;
            pos_x += 40;
            pos_y += 0;

            rl.drawTexture(texture, pos_x + pos.x_offset, pos_y + pos.y_offset, rl.Color.white);
        }
    }
}

To make it easier to reproduce / experiment, I’ve also added this code (including the Devanagari ttf that I’m using) to a small github repo: GitHub - lhk/zig_render_devanagari at adding_raylib The repo also has a zig.build and zig.build.zon :slight_smile:

1 Like

Font metrics are complicated and it has been a while since I have done glyph stuff (I also never done shaping only primitive glyph drawing), but here are some hints (which might not be totally accurate).

pos.x_advance, pos.y_advance, pos.x_offset, pos.y_offset are in f26dot6 font units.
That unit is basically an i32 divided by 64 (2^6 == 64).

Another part that is missing from your example is to use more of the glyph metrics this example also uses bearing_x and bearing_y:

I think those are used to reposition the glyphs in a way so that they align correctly in relation to each other more info here FreeType Glyph Conventions | Glyph Metrics.

I think if you reverse engineer / try to copy the math of that example, you should at least be closer towards a correct text layout solution:

Good luck, so far I haven’t done this myself, but that example looks like it might get you closer towards good results.

2 Likes

@Sze thank you for the links and the advice. That’s really quite helpful.

I’ve adjusted my code and the output looks a bit better. It’s not clear to me how to get the bearing of the glyph though.
I don’t think that field is exposed by mach-freetype : Code search results · GitHub

And after reading through the resources you’ve shared, it seems that the bearing is only used for individual glyph offsets. Something like that is definitely needed, to get the accents on top of the letters positioned correctly. But I also think the overall spacing is still wrong:
image

Looks a bit like the Logo of a metal band :slight_smile:

            const x_advance = @as(f32, @floatFromInt(pos.x_advance));
            const y_advance = @as(f32, @floatFromInt(pos.y_advance));
            const x_offset = @as(f32, @floatFromInt(pos.x_offset));
            const y_offset = @as(f32, @floatFromInt(pos.y_offset));

            pos_x += x_advance / 64;
            pos_y += y_advance / 64;

            const draw_x: i32 = @intFromFloat(pos_x + x_offset / 64);
            const draw_y: i32 = @intFromFloat(pos_y + y_offset / 64);

            // ToDo: how do I get the bearing of the glyph?

            rl.drawTexture(texture, draw_x, draw_y, rl.Color.white);
2 Likes

I have currently this:

const fs = 40;
const fs_pt = fs * 64;
// try ft_face.setCharSize(fs, fs, 120, 120);
try ft_face.setCharSize(0, fs_pt, 0, 0);
...

const b = rl.Vector2{ .x = 100, .y = 100 };
var p = b;

for (glyphInfos, glyphPositions) |info, pos| {
  ...

  const xa = @as(f32, @floatFromInt(pos.x_advance)) / 64;
  const ya = @as(f32, @floatFromInt(pos.y_advance)) / 64;
  const xo = @as(f32, @floatFromInt(pos.x_offset)) / 64;
  const yo = @as(f32, @floatFromInt(pos.y_offset)) / 64;
  
  const metrics = glyph.metrics();
  const bearing_x = @as(f32, @floatFromInt(metrics.horiBearingX)) / 64;
  const bearing_y = -@as(f32, @floatFromInt(metrics.horiBearingY)) / 64;
  const x0 = p.x + xo + bearing_x;
  const y0 = @floor(p.y + yo + bearing_y);
  
  const p0 = rl.Vector2{ .x = x0, .y = y0 };
  rl.drawTextureEx(texture, b.add(p0), 0, 1, rl.Color.white);
  p = p.add(rl.Vector2{ .x = xa, .y = ya });
}

Which looks like this:
Screenshot_2024-08-16_17-41-40

I think the main problem is that there is still some font size scaling missing, maybe it would make sense to switch to mesh construction to get the two examples to be closer in terms of implementation.
Would probably also make sense in terms of better performance.

I think you can construct manual meshes with raylib, at least I heard people do it, so far I have only loaded some external meshes.
Also creating a texture atlas would make sense, currently it just leaks textures like crazy. :laughing:

This advance needs to be applied to the pos_x and pos_y accumulators after you have used the values of pos_x and pos_y because the advance is for the next glyph not for the current one, so basically shift those lines below the draw_x and draw_y lines.

I might fiddle more around with it another time, but hopefully that helps.

1 Like

Oh wow, that’s a huge help. Thank you so much :slight_smile:

Yea, the textures leaking :see_no_evil:, I did notice that the memory usage of this little program keeps climbing and climbing …
And if I add rl.unloadTexture then it stays constant. But it also no longer draws the texture. It’s not clear to me what’s happening there. Maybe it’s unloading the texture too early? Overall that felt like an orthogonal issue so I thought I’d first try to figure out the font layout.

It’s not clear to me what you mean by manual meshes, but I’ll see if I can find something with that keyword. Texture Atlas sounds good. So I guess I’d do something like a Hashmap from glyph codepoint to texture and see if that’s already populated? That should be doable, I’ll take a look.

I really really appreciate the time you took to help here. Thanks!

1 Like

Yes it is an orthogonal issue.

Mesh is basically a buffer of vertex data that describes geometry, the idea would be to generate a quad (or 2 triangles depending on api) for every glyph and then the vertices of that would get texture coordinates that map to the texture atlas.

Instead you would have a Hashmap from glyph codepoint to a small struct that describes the rectangle within a bigger texture that stores all the needed glyphs.
So instead of having multiple textures you would only have one texture that contains all the glyphs.

But for that you also need to implement or use a texture atlas generator / packer. Probably there are already written ones for zig. I also had created one before I started using zig, might port that to zig some time.

The main advantage of mesh + atlas would be that you use fewer draw calls, fewer texture switches and reuse glyphs and put less pressure on raylib’s internal batching mechanism.

That internal batcher can make some things work quite well performance wise, but explicit manual batching techniques like using a mesh and generating atlases just know more and thus are very likely to result in better performance.

1 Like

I’ve added a lookup table to reuse textures. This allows me to not allocate new Textures in each loop iteration and I think solves the memory leak (Though I guess I still need to go through the Hashmap values and unload them at the program exit).

Maybe more interestingly: After playing with the numbers in setCharSize the font now looks like this:
image

That’s for this configuration:

    // ToDo: understand these magic numbers
    try ft_face.setCharSize(0, 5000, 9, 35);

The texture lookup looks like this:

    var textureHashMap = std.hash_map.AutoHashMap(u32, rl.Texture2D).init(std.heap.page_allocator);
    defer textureHashMap.deinit();

...
            const maybeTexture = textureHashMap.get(info.codepoint);
            var texture: rl.Texture2D = undefined;
            if (maybeTexture == null) {

            // create new texture from glyph bitmap
            // ...

So maybe now all that’s left is figuring out how to parametrize the face correctly? I saw there’s also a call for setting pixel sizes: freetype2 - Setting Character Size in FreeType - Stack Overflow

1 Like

@lhk So I ended up spending a bit more time on this, basically fusing your, my and the code from the example I linked.

There are probably still a bunch of leaks, things that don’t get deinit-ed correctly (I didn’t really look at the freetype and harfbuzz wrappers in detail to figure out how they are meant to be used).

Also there is a magic scale value (I don’t quite know why it is needed) would have to analyze the math a bit more and double check some more documentation, or alternatively step through it with a debugger.

But I think it is a good start, I think with a bit more refinement it could be turned into a small utility library for rendering text.

const std = @import("std");
const freetype = @import("mach-freetype");
const harfbuzz = @import("mach-harfbuzz");
const font_assets = @import("font-assets");

const rl = @import("raylib");
const rg = @import("raygui");

// based on:
// https://ziggit.dev/t/rendering-text-with-harfbuzz-freetype/5636
// and https://github.com/tangrams/harfbuzz-example/blob/1305db5b4bd7034d4cd5adb2c5cbfaa02e0e095e/src/hbshaper.h#L108-L113

const Shaper = struct {
    const Textures = std.hash_map.AutoHashMap(u32, rl.Texture2D); // TODO replace with TextureAtlas
    //
    allocator: std.mem.Allocator,
    ftlib: *const freetype.Library,

    face: freetype.Face,

    hb_face: harfbuzz.Face,
    hb_font: harfbuzz.Font,
    hb_buffer: harfbuzz.Buffer,

    textures: Textures,
    data: std.MultiArrayList(struct { // TODO replace with mesh
        glyph: u32,
        pos: rl.Vector2,
    }) = .{},

    index: u32 = 0,

    pub fn init(allocator: std.mem.Allocator, ftlib: *const freetype.Library, face: freetype.Face) !Shaper {
        const hb_face = harfbuzz.Face.fromFreetypeFace(face);
        const hb_font = harfbuzz.Font.init(hb_face);
        return .{
            .allocator = allocator,
            .ftlib = ftlib,
            .face = face,
            .hb_face = hb_face,
            .hb_font = hb_font,
            .hb_buffer = harfbuzz.Buffer.init() orelse return error.HarfbuzzBufferFail,
            .textures = Textures.init(allocator),
        };
    }
    pub fn deinit(self: *Shaper) void {
        self.release();
        self.textures.deinit();
        // TODO deinit freetype and harfbuzz handles
    }

    fn release(self: *Shaper) void {
        self.hb_buffer.reset();

        var it = self.textures.valueIterator();
        while (it.next()) |texture| rl.unloadTexture(texture.*);
        self.textures.clearRetainingCapacity();

        self.data.shrinkAndFree(self.allocator, 0);
    }

    pub fn setText(self: *Shaper, direction: harfbuzz.Direction, script: harfbuzz.Script, text: []const u8) !void {
        self.release();

        self.hb_buffer.setDirection(direction);
        self.hb_buffer.setScript(script);
        self.hb_buffer.addUTF8(text, 0, @intCast(text.len));
        self.hb_font.shape(self.hb_buffer, null);

        // glyph infos and positions, that's what we need to render the text
        const glyphInfos = self.hb_buffer.getGlyphInfos();
        const glyphPositions = self.hb_buffer.getGlyphPositions() orelse return;

        var p = rl.Vector2.zero();
        for (glyphInfos, glyphPositions) |info, pos| {
            const index = info.codepoint;
            try self.face.loadGlyph(index, .{ .render = true });
            const glyph = self.face.glyph();

            const scale = 5; // TODO can we eliminate this magic value? This either shouldn't be necessary or be calculated based font size or something else
            const xa = scale * @as(f32, @floatFromInt(pos.x_advance)) / 64.0;
            const ya = scale * @as(f32, @floatFromInt(pos.y_advance)) / 64.0;
            const xo = @as(f32, @floatFromInt(pos.x_offset)) / 64;
            const yo = @as(f32, @floatFromInt(pos.y_offset)) / 64;
            const advance = rl.Vector2{ .x = xa, .y = ya };

            const bm = glyph.bitmap();
            const maybe_buffer = bm.buffer();
            if (maybe_buffer == null) {
                p = p.add(advance);
                continue;
            }

            const entry = try self.textures.getOrPut(index);
            if (!entry.found_existing) {
                entry.value_ptr.* = rl.loadTextureFromImage(.{
                    .width = @intCast(bm.width()),
                    .height = @intCast(bm.rows()),
                    .mipmaps = 1,
                    .format = rl.PixelFormat.pixelformat_uncompressed_grayscale,
                    .data = @constCast(maybe_buffer.?.ptr),
                });
            }

            const metrics = glyph.metrics();
            const bearing_x = @as(f32, @floatFromInt(metrics.horiBearingX)) / 64;
            const bearing_y = -@as(f32, @floatFromInt(metrics.horiBearingY)) / 64;
            const x0 = p.x + xo + bearing_x;
            const y0 = @floor(p.y + yo + bearing_y);

            const p0 = rl.Vector2{ .x = x0, .y = y0 };
            try self.data.append(self.allocator, .{ .glyph = index, .pos = p0 });

            std.debug.print("index: {} pos: {}\n", .{ index, p0 });

            p = p.add(advance);
        }
    }

    pub fn draw(self: *Shaper, start_pos: rl.Vector2) void {
        const s = self.data.slice();

        if (rl.isKeyPressed(.key_left)) self.index -|= 1; // debugging feature
        if (rl.isKeyPressed(.key_right)) self.index +|= 1;

        for (s.items(.glyph), s.items(.pos), 0..) |glyph, pos, i| {
            const c = if (i == self.index) rl.Color.green else rl.Color.white;
            if (i == self.index) {
                std.debug.print("texture: {}\n", .{self.textures.get(glyph).?});
            }

            rl.drawTextureEx(self.textures.get(glyph).?, start_pos.add(pos), 0, 1, c);
        }
    }
};

pub fn main() !void {
    const screenWidth = 800;
    const screenHeight = 800;
    rl.initWindow(screenWidth, screenHeight, "devanagari text");
    defer rl.closeWindow();
    rl.setTargetFPS(60);

    // Normally I use the GPA allocator, but it seems incompatible with raylib.
    // See further below for a comment on this issue.
    // var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    // const allocator = gpa.allocator();
    // defer {
    //     _ = gpa.deinit();
    // }
    const allocator = std.heap.c_allocator;

    const text = "मेरा नाम लार्स है";

    // set up freetype face
    const lib = try freetype.Library.init();
    defer lib.deinit();
    const ft_face = try lib.createFaceMemory(@embedFile("NotoSerifDevanagari_Condensed-Bold.ttf"), 0);

    const fs = 80;
    const fs_pt = fs * 64;
    try ft_face.setCharSize(0, fs_pt, 72, 72);

    var shaper = try Shaper.init(allocator, &lib, ft_face);
    defer shaper.deinit();

    try shaper.setText(.ltr, .devanagari, text);

    while (!rl.windowShouldClose()) {
        rl.beginDrawing();
        defer rl.endDrawing();
        rl.clearBackground(rl.Color.black);
        rl.beginBlendMode(rl.BlendMode.blend_additive);
        defer rl.endBlendMode();

        shaper.draw(rl.Vector2{ .x = 100, .y = 200 });
    }
}

Screenshot_2024-08-16_22-55-17

3 Likes

I think with a bit more refinement it could be turned into a small utility library for rendering text.

Obligatory shout-out to a library I wrote a while back: zig-fontmanager/font_manager.zig at main · mlugg/zig-fontmanager · GitHub

It definitely won’t compile on latest Zig, needs some tweaks, but it did manage font atlas textures in a quite pleasing way and worked excellently on English text – I never tested it on more complex scripts, I wonder if it works.

EDIT: wow, I just looked at the build script, that code is old old

2 Likes

Hey all, I got a bit distracted with various stuff and haven’t found the time to reply so far.

But I wanted to say thanks!
@Sze your help was invaluable.
@mlugg that is a really cool little library, thanks for mentioning it.

I’ve marked your last post as an answer Sze, I think for people looking for a solution this will be good enough to get them going.
I do want to look more at those magic numbers in font size etc eventually. If I find something principled I’ll post it here.

1 Like