Emscripten IDBFS issue with raylib-zig and zig 0.16

First of all, I have to apologize for posting here, as this forum might not be the appropriate place to post about this issue, but I’m not sure where the issue would be between zig, zemscripten, raylib-zig and emsdk. The input I am looking for is mostly about if this is a known issue, if anybody researched something similar.

I’m making a wasm game using raylib-zig and zig 0.15.2. For the save game, I use emscripten’s IDBFS to be able to use the IndexDB through emscripten file API, to avoid needing something platform specific, so I can do the bulk of development on on Windows and have wasm as the target platform with more limited debugging.

When I updated to 0.16.0 (when it was master, and later as the main release) I could get most of it to work pretty smoothly, except for the persistent storage (IDBFS). I didn’t get more success with more recent master build.

I created a minimal program to exercise the write to IndexDB browser storage and tried to bisect with the raylib-zig dependency and got here. The latest working version I got is right before upgrading to 0.16.0:

zig fetch --save git+https://github.com/raylib-zig/raylib-zig#aa9ee05f2246b813f75206bf817c9fbdfcb06101

After upgrading to 0.16.0, with:

zig fetch --save git+https://github.com/raylib-zig/raylib-zig#8e04b7098af58c36d9b8cb9ed84eca3d47f1d048

I tried to keep the build.zig as close as I could to the readme documentation for wasm and I used the basic window example as a basis for the app: it’s only drawing text to show it’s doing raylib’y stuff, and saving a file every 3 seconds (because syncing the FS is async, so it’s available a bit after booting). To observe it, use the browser debugger (console and storage).

The obvious problem is that in the 0.16.0 version, the emscripten’s js blob doesn’t contain any dbfs symbol (IDBFS only appears in unexported symbol). It is puzzling, because the -lidbfs.js is processed (e.g. I get a build error with -ldbfss.js). In the working 0.15.2 version, IDBFS appears 39 times (in filesystem related part, like mount, rename) in the js blob.

Here are my project files for the working version

src/main.zig: save a file every 3 seconds. Logs to console and change the text color every 3 when save time happens.

const rl = @import("raylib");
pub fn main() anyerror!void {
    rl.initWindow(360, 360, "file write test");
    defer rl.closeWindow();
    rl.setTargetFPS(15);
    var save_after: i32 = 3;
    while (!rl.windowShouldClose()) {
        if (@as(f32, @floatFromInt(save_after)) < rl.getTime()) {
            rl.traceLog(@enumFromInt(3), "Saving test_file.sav at %d", .{save_after});
            if (rl.saveFileData("test_file.sav", @constCast("<== test content ==>"))) {
                rl.traceLog(@enumFromInt(3), "Saved test_file.sav at %d", .{save_after});
            }
            save_after += 3;
        }
        rl.beginDrawing();
        defer rl.endDrawing();
        rl.clearBackground(.dark_gray);
        rl.drawText("Check your browser's IndexedDB\nFirefox's Storage or \nChromium's Application->Storage)", 5, 140, 20, if (save_after & 1 == 0) .light_gray else .gray);
    }
}

raylib/src/saveshell.html pre-run mounts /saves to IDBFS and auto-persists

<!doctype html>
<html>
    <body>
        <canvas class=emscripten id=canvas oncontextmenu=event.preventDefault() tabindex=-1></canvas>
        <p id="output" />
        <script>
            var Module = {
                preRun: () => {
                    // Set up the saves folder with IDBFS
                    FS.mkdir('/saves')
                    FS.mount(IDBFS, {autoPersist: true}, '/saves')
                    FS.chdir('/saves')
                    FS.syncfs(true, err => {
                        if (err) {
                            console.log(err)
                        }
                    })
                },
                print: (function() {
                    return function(text) {
                        if (arguments.length > 1) text = Array.prototype.slice.call(arguments).join(' ');
                        console.log(text);
                    };
                })(),
                canvas: (function() {
                    var canvas = document.getElementById('canvas');
                    return canvas;
                })()
            };
        </script>
        {{{ SCRIPT }}}
    </body>
</html>

build.zig adds the emcc_flags.put(“-lidbfs.js”, {})

const std = @import("std");
const rlz = @import("raylib_zig");

pub fn build(b: *std.Build) !void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    const raylib_dep = b.dependency("raylib_zig", .{
        .target = target,
        .optimize = optimize,
    });
    const raylib = raylib_dep.module("raylib");
    const raylib_artifact = raylib_dep.artifact("raylib");

    const exe_mod = b.createModule(.{
        .root_source_file = b.path("src/main.zig"),
        .target = target,
        .optimize = optimize,
    });
    exe_mod.addImport("raylib", raylib);

    const run_step = b.step("run", "Run the app");

    //web exports are completely separate
    if (target.query.os_tag == .emscripten) {
        const emsdk = rlz.emsdk;
        const wasm = b.addLibrary(.{
            .name = "minimal-raylib-wasm",
            .root_module = exe_mod,
        });

        const install_dir: std.Build.InstallDir = .{ .custom = "web" };
        var emcc_flags = emsdk.emccDefaultFlags(b.allocator, .{
            .optimize = optimize,
            .asyncify = true, // except for the examples/core/basic_window_web.zig example, which is false
        });
        emcc_flags.put("-lidbfs.js", {}) catch unreachable;
        emcc_flags.put("-sINITIAL_MEMORY=4194304", {}) catch unreachable;

        const emcc_settings = emsdk.emccDefaultSettings(b.allocator, .{
            .optimize = optimize,
        });

        const shell = b.path("raylib/src/saveshell.html");

        const emcc_step = emsdk.emccStep(b, raylib_artifact, wasm, .{
            .optimize = optimize,
            .flags = emcc_flags,
            .settings = emcc_settings,
            .shell_file_path = shell,
            .install_dir = install_dir,
            .embed_paths = &.{.{ .src_path = "resources/" }},
        });
        b.getInstallStep().dependOn(emcc_step); // not in the example version

        const html_filename = try std.fmt.allocPrint(b.allocator, "{s}.html", .{wasm.name});
        const emrun_step = emsdk.emrunStep(
            b,
            b.getInstallPath(install_dir, html_filename),
            &.{},
        );

        emrun_step.dependOn(emcc_step);
        run_step.dependOn(emrun_step);
    } else {
        const exe = b.addExecutable(.{
            .name = "minimal-raylib-wasm",
            .root_module = exe_mod,
        });
        b.installArtifact(exe);

        const run_cmd = b.addRunArtifact(exe);
        run_cmd.step.dependOn(b.getInstallStep());

        run_step.dependOn(&run_cmd.step);
    }
}

build.zig.zon

.{
    .name = .minimal_raylib_wasm,
    .version = "0.0.0",
    // raylib zig: aa9ee05f2246b813f75206bf817c9fbdfcb06101
    // zig fetch --save git+https://github.com/raylib-zig/raylib-zig#aa9ee05f2246b813f75206bf817c9fbdfcb06101
    // zig fetch --save=emsdk git+https://github.com/emscripten-core/emsdk#4.0.9
    .dependencies = .{
        .emsdk = .{
            .url = "git+https://github.com/emscripten-core/emsdk?ref=4.0.9#3bcf1dcd01f040f370e10fe673a092d9ed79ebb5",
            .hash = "N-V-__8AAJl1DwBezhYo_VE6f53mPVm00R-Fk28NPW7P14EQ",
        },
        .raylib_zig = .{
            .url = "git+https://github.com/raylib-zig/raylib-zig#aa9ee05f2246b813f75206bf817c9fbdfcb06101",
            .hash = "raylib_zig-5.6.0-dev-KE8RECFQBQBn0BrEyEyK5NST461oRzpk6XeGeF9qBU2M",
        },
        .raylib = .{
            .url = "git+https://github.com/raysan5/raylib#9f831428e6be0eba8762a154e3e9139d4f071970",
            .hash = "raylib-5.6.0-dev-whq8uL592gQ782dF95oci4S61qrNVYK3MhpUpNwFgZ1J",
        },
        .raygui = .{
            .url = "git+https://github.com/raysan5/raygui#9cdfec460b43a17264af3c181c46f62bf107ac17",
            .hash = "N-V-__8AALUbbwDKkSH4nbf3Ml_dTWo9qbELvle5i9eQZMuo",
        },
    },
    .fingerprint = 0x104a75e13622d92b, // Changing this has security and trust implications.
    .minimum_zig_version = "0.15.2",

    .paths = .{
        "build.zig",
        "build.zig.zon",
        "src",
    },
}

For the 0.16.0 version, fetch the 0.16 compatible raylib, and work around the std lib problem.

My next steps are:

  • try to generate a minimum reproduction with only zemscripten, and bisect that. From there I might be closer to the actual problem.
  • wait 0.17.0 and try again at that point
  • sidestep emscripten FS entirely and have some weird platform specific file-like storage.

But I wanted to know if I’m alone experiencing this issue.

1 Like

I could not reproduce the issue, IDBFS works on both 0.15 and 0.16. I however needed to make the following addition/changes to your repro:

  • Create the directory resources and an empty file inside of it.
  • For 0.16:
    • Run zig fetch --save git+https://github.com/raylib-zig/raylib-zig#97be2c7ae646a2f09856604b138c427c0af1b952 (the currently latest commit on the devel branch)
    • Remove the switch prongs that call takeown on Windows from all zemscripten build.zigs inside zig-pkg because they didn’t work on my system for some reason.
    • Add
      const std = @import("std");
      pub const std_options_debug_io = std.Io.failing;
      pub const panic = std.debug.no_panic;
      
      to main.zig.

I would suggest that you double-check that your browser hasn’t cached some earlier work-in-progress build of your repro. Other things that could cause issues:

  • By default, Emscripten may omit including the filesystem component if it detects that you aren’t using filesystem APIs (implementation details). raylib’s SaveFileData opens a file for writing so this should be the case for your repro, but you might want to be aware of this behavior if you’ve tested with e.g. an empty main function. You can pass -sFORCE_FILESYSTEM to emcc to disable this behavior.
  • If you’re passing --closure 1 to emcc to minify JavaScript and eliminate dead code, you want to make sure that symbols that are only referenced from your HTML shell or --pre-js/--post-js aren’t eliminated. For example, you might need to use Module['preRun'] = () => { FS['mount'](IDBFS, ...) }, as using indexers instead of dot access is the way you instruct Closure that a property must not be trimmed.

If the problem persists it’s almost certainly with how raylib/raylib-zig/emsdk/zemscripten are implemented, not Zig itself, so you will probably have better luck if you ask them for help directly.

As an aside, I personally really dislike how packages like raylib use emsdk (see my replies to this thread) and would advice against doing what they do. Building for Emscripten is very fickle and it’s easy to have the Emscripten cache end up in an inconsistent state, leading to confusing issues and inconsistencies that will be difficult for most users to untangle unless they are deeply familiar with Emscripten internals. It’s also a problem that things like emccStep hide important details about how the Emscripten toolchain is invoked and make it more difficult than necessary for users to customize or debug their emcc invocations when problems arise. Running emcc manually requires a few more lines of boilerplate code upfront, but is so much easier for a reader (= you in 3 months) to understand and debug.

3 Likes

I’m very thankful for the help!

While I was writing a fairly long post about everything you suggested that I tried and didn’t work, I noticed that the SDK step was setting the environment variable to a zig-pkg on one drive, but running the command on another drive: the path resolved the junction but not the calling method. The reason is that I was executing from under the link location (C:/devel) but that folder is a junction to under my user (~/devel) somewhere on the G: drive.

If I had to guess, I would say it’s a problem in the way emcc.bat is called and is resolving its directory, but it explains why there is a difference between 0.15 and 0.16: in 0.15, emcc.bat was called from the zig env’s .global_cache_dir, which didn’t have any kind of symlink or reparse point involved, but 0.16.0’s zig-pkg’s emcc.bat was sitting 6 levels under a symlink.

Sorry for your time, it was a case of doesn’t work on my machine.

1 Like

I’m just thinking, it would probably be best if the zig build system could always resolve directory symlink in LazyPath and such, maybe a command line option.

edit: just for fun, I tried subst, it fails more steps, but has the merit of having an error message instead of just not producing the expected output:

  • the takeown call completely fail
  • we get explicit failure like ValueError: path is on mount 'N:', start on mount 'G:' from python