Wasm32-emscripten now requires Emscripten filesystem emulation (side effect of new IO?)

Brainstorming
1

I just noticed a new-ish emcc-link-problem when building the sokol-zig samples (GitHub - floooh/sokol-zig: Zig bindings for the sokol headers (https://github.com/floooh/sokol) · GitHub) for wasm32-emscripten (via zig build -Dtarget=wasm32-emscripten examples) using the latest Zig nightly.

This started to fail in the emcc linker step with a missing symbol required by the accept4 syscall:

error: undefined symbol: $SOCKFS (referenced by $getSocketFromFD, referenced by __syscall_accept4, referenced by root reference (e.g. compiled C/C++ code))
warning: To disable errors for undefined symbols use `-sERROR_ON_UNDEFINED_SYMBOLS=0`

TL;DR: this says that even programs that don’t call socket functions now depend on a socket syscall for accept().

This is a linker error because I’m linking with the emcc option -sNO_FILESYSTEM=1 which essentially disables the Emscripten POSIX IO emulation (e.g. the problem is easy to fix by just not passing this option to the linker - but this increases the ‘binary size’).

It made me think though: why does a Zig program that doesn’t even use socket functionality all of the sudden pull in the socket accept() function, when Zig has a ‘lazy’ compilation model which only builds reachable code?

Is this a side effect of the new IO being a virtual method interface which essentially kills dead-code-elimination?

If that’s the reason, will this problem (and IMHO it’s a pretty big problem) even be fixable? It would kinda suck if code that’s never going to be called is linked into each and every Zig program.

If this is not fixable in the compiler, are there plans for a ‘minimal/embedded IO’ which doesn’t require files and sockets? (or maybe a ‘modular IO’ that can be configured via build options - e.g. similar to how I can pass -sNO_FILESYSTEM=1 to Emscripten which promises that I will not call any code which depends on POSIX IO).

PS: resulting size comparison (uncompressed, in bytes):

Zig 0.15.2 with Emscripten -sNO_FILESYSTEM=1 and release=small:

clear.js: 23620 bytes
clear.wasm: 33443 bytes

Current Zig nightly with Emscripten filesystem emulation enabled, release=small:

clear.js: 58078 bytes
clear.wasm: 84700 bytes

Even though it’s “just” a couple dozen kbytes, for small programs this overhead is quite substantial…

4 Likes
2

Indeed, even standard hello world for x86_64-linux seems to bring ChaCha and other goodies. I don’t think zig can lazily reference struct fields, perhaps restricted function pointers can somehow figure out all the dead code? Not sure how the core team aims to tackle this, but for now the alternative is either to not use the Io interface or implement your own or use 0.15.2.

3 Likes
3

I saw this PR mentioning lazy field analysis, could this help?

4

yes, i believe one of the motivations for the newly-added lazy field analysis in that PR is exactly for ease of use of std.Io as a namespace type without necessarily pulling in all the code from std.Io as an instantiable type.

of course, if you use a std.Io then we’re back to the problem again

4 Likes
5

I don’t think so, as on source level Io is a interface and goes through *anyopaque pointer. It would require information from some sort of optimization pass (that can also figure out which type and storage location the pointer points to at that point of code at compile time).

1 Like
6

That pr did say “lazy field analysis”, but it did not mean that in the way @Cloudef suggested.

That pr meant that if you accessed a declaration it would not analyse the fields unless necessary, and gives the example that previously using std.Io.Writer resulted in the analysis of the std.Io type, including its fields, which includes its vtable and causing binary bloat as a consequence.

After that pr, using std.Io.Writer will no longer analyse std.Io.Vtable.
But an instance of std.Io will still require all the fields and vtable to be analysed.

6 Likes
7

There is an inherent complexity in optimizing unused fields of a structure. Whether a certain field is used is related to the size of the structure or the presence of other fields, and optimizing these fields will affect these control flows.

const Vtb = struct {
    aF: *const fn (*anyopaque) void,
    bF: *const fn (*anyopaque) void,
};

const Interface = struct {
    ptr: *anyopaque,
    vtb: Vtb,
    pub fn do(self: Interface) void {
        if (@sizeOf(Vtb) > @sizeOf(usize)) {
            self.vtb.aF(self.ptr);
        } else {
            self.vtb.aF(self.ptr);
            self.vtb.bF(self.ptr);
        }
    }
};

If we start with optimizing the struct fields of an interface, we might need to design a special kind of struct, whose comptime @sizeOf and field information cannot be obtained, and only its post-comptime related information can be obtained (similar to the not-yet-implemented @stackSize). This may essentially implement the interface at the language level.

Making the interface a configurable generic might be a solution I can accept more.

pub const Config = struct {
    aF: bool,
    bF: bool,
};

pub fn Interface(comptime config: Config) type {
    return struct {
        ptr: *anyopaque,
        vtb: Vtb,
        pub const Vtb = struct {
            aF: if (config.aF) *const fn (*anyopaque) void else void,
            bF: if (config.bF) *const fn (*anyopaque) void else void,
        };
        pub fn aF(self: @This()) void {
            if (!config.aF) @compileError("This interface function is configured to be unavailable.");
            self.vtb.aF(self.ptr);
        }
        pub fn bF(self: @This()) void {
            if (!config.bF) @compileError("This interface function is configured to be unavailable.");
            self.vtb.bF(self.ptr);
        }
    };
}

Another possibility is to follow the pattern of std_options and let users configure it in the root module through overrides.

const root = @import("root");

/// Stdlib-wide options that can be overridden by the root file.
pub const options: IoOptions = if (@hasDecl(root, "io_options")) root.io_options else .{};

pub const IoOptions = struct {
    ...,
    net_support: bool = true,
    ...,
};

pub const Io = struct {
    userdata: ?*anyopaque,
    vtable: *const VTable,
    pub const VTable = struct {
        ...,
        netAccept: if (options.net_support) *const fn (?*anyopaque, server: net.Socket.Handle) net.Server.AcceptError!net.Stream else void,
        ...,
    };
};

Besides the vtb interface, another pattern of dynamic polymorphism is the tagged union. Although it is considered difficult to customize types, perhaps a similar approach can be used to make the types supported by the tagged union configurable.

8

The standard library exposes a toggle for networking in std.Options

    /// Allows disabling networking in std.Io implementations.
    networking: bool = true,

Also looks like many more functions will be moved to the batch/operate API; recently netReceive was moved there.

About binary size, you can ducktype the VTable if needed and lazy analysis will still work as expected, but that is currently not available in master so you would have to maintain a fork as I do (note: it does more than just allowing to override the VTable)

3 Likes
9

Tracking issue is https://codeberg.org/ziglang/zig/issues/31421

Yes, it’s a side effect of using std.Io - all the functions of the chosen implementation are compiled, even if unused. This will be a known regression in the 0.16.x release series.

There’s a mitigation, set .networking = false, in your std.Options.

In a future release cycle, we’ll work towards eliminating the dead code and making the mitigation unneeded.

20 Likes