Confusion on file Reader + JSON parsing errors

Explain
1

I was banging my head against this issue for about 2 hours, then got it working just trying a slightly different approach that I saw with a Github search.

The problem is - both solutions seem identical to me, so I’m completely bewildered as to why one fails and one succeeds. Any help understanding what the difference is would be greatly appreciated.

Failing example:

var buffer: [1024]u8 = undefined;
var reader = file.reader(&buffer);
var reader_io = reader.interface;

var json_reader = std.json.Reader.init(allocator, &reader_io);
defer json_reader.deinit();

const json = std.json.parseFromTokenSource(ScenesJson, allocator, &json_reader, .{}) catch |e| {...}

Succeeding example:

var buffer: [1024]u8 = undefined;
var reader = file.reader(&buffer);

var json_reader = std.json.Reader.init(allocator, &reader.interface);
defer json_reader.deinit();

const json = std.json.parseFromTokenSource(ScenesJson, allocator, &json_reader, .{}) catch |e| {...}

The failing example produces the following error at runtime with a Debug build:

panic: switch on corrupt value
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/json/Scanner.zig:1276:5: 0x7f654b55f553 in expectByte (std.zig)
    return error.BufferUnderrun;
    ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/json/Scanner.zig:1300:5: 0x7f654b55f616 in skipWhitespaceExpectB
yte (std.zig)
    return self.expectByte();
    ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/json/Scanner.zig:325:25: 0x7f654b560537 in next (std.zig)
                switch (try self.skipWhitespaceExpectByte()) {
                        ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/fs/File.zig:1344:17: 0x7f654b55c275 in readVec (std.zig)
        switch (r.mode) {
                ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/Io/Reader.zig:1063:56: 0x7f654b5331c9 in fillUnbuffered (std.zig
)
    while (r.end < r.seek + n) _ = try r.vtable.readVec(r, &bufs);
                                                       ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/Io/Reader.zig:1049:26: 0x7f654b51b2ff in fill (std.zig)
    return fillUnbuffered(r, n);
                         ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/Io/Reader.zig:473:15: 0x7f654b572010 in peekGreedy (std.zig)
    try r.fill(n);
              ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/json/Scanner.zig:1736:45: 0x7f654b5697fd in refillBuffer (std.zi
g)
        const input = self.reader.peekGreedy(1) catch |err| switch (err) {
                                            ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/json/Scanner.zig:1714:42: 0x7f654b559078 in next (std.zig)
                    try self.refillBuffer();
                                         ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/json/static.zig:334:49: 0x7f654b5579d9 in innerParse__anon_23202
 (std.zig)
            if (.object_begin != try source.next()) return error.UnexpectedToken;
                                                ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/json/static.zig:149:33: 0x7f654b5555e3 in parseFromTokenSourceLe
aky__anon_23032 (std.zig)
    const value = try innerParse(T, allocator, scanner_or_reader, resolved_options);
                                ^
/home/adub/.local/share/mise/installs/zig/0.15.2/lib/std/json/static.zig:116:49: 0x7f654b552ccb in parseFromTokenSource__
anon_22864 (std.zig)
    parsed.value = try parseFromTokenSourceLeaky(T, parsed.arena.allocator(), scanner_or_reader, options);
                                                ^
/home/adub/src/zscene/src/read.zig:101:47: 0x7f654b550deb in readScenesCreate (zscene.zig)
    const json = std.json.parseFromTokenSource(ScenesJson, allocator, &json_reader, .{}) catch |e| {

Essentially, the main difference is that the failing example stores the Io interface in a var variable (since const fails to build due to the Reader.init requiring a non-const Io interface), while the succeeding example passes the Io interface reference inline.

I’m a bit bewildered by why there’s a difference, since it seems like the Io interface lives in the same scope in cases.

2

The reader/writer interfaces must be used through pointers to the fields.

In the failing code you are copying it out of the field, then using it.

This is because they do not store a pointer to the implementation state, so in order for the vtable functions to access the implementation state they do some pointer magic on the interface pointer, that assumes it is pointing to a specific field in a specific type.

It has no way currently to know if the interface pointer is pointing where it expects or not, it will treat it as the implementation state regardless, if it is not then it is interpreting arbitrary memory as though it were and that memory could have invalid values which is the error you have encounted.

4 Likes
3

Got it, thanks for the detailed explanation.

I need to pay a bit more attention to how assignment works in Zig, as I get caught by things like this more often than I’d like.

4

It has no way currently to know if the interface pointer is pointing where it expects or not, it will treat it as the implementation state regardless, if it is not then it is interpreting arbitrary memory as though it were and that memory could have invalid values which is the error you have encounted.

Wonder if this is something that can be done? When I was learning Zig I also forgot the & and it had me going in circles for hours. Zig tries to stay out of your way, which I love, but I wonder if this could be caught at comptime/runtime more cleanly, probably with some code in the stdlib?

5

it can and will be caught at runtime add safety checks for pointer casting · Issue #2414 · ziglang/zig · GitHub

At compile time would be ideal, but no proposal for that has been good enough to be accepted.

1 Like