Resizing with PageAllocator

Help
1

Here is the excerpt from PageAllocator.zig:

    if (new_size_aligned < buf_aligned_len) {
        const ptr = buf_unaligned.ptr + new_size_aligned;
        // TODO: if the next_mmap_addr_hint is within the unmapped range, update it
        os.munmap(@alignCast(ptr[0 .. buf_aligned_len - new_size_aligned]));
        return true;
    }

    // TODO: call mremap
    // TODO: if the next_mmap_addr_hint is within the remapped range, update it
    return false;

Does it mean that resizing up is impossible now with PageAllocator? Shouldn’t there be an issue for that?

2

I would suggest that you write a failing test case to show what you think isn’t working (or just to confirm that there’s something wrong).

2 Likes
3

I don’t have that reflex yet, thanks for the suggestion. Here it is:

const std = @import("std");

test "PageAllocator resize" {
  const page_allocator = std.heap.page_allocator;
  var buf = try page_allocator.alloc(i32, 100);
  try std.testing.expect(page_allocator.resize(buf, 100 + std.mem.page_size) == true);
}

This test fails with:

Test [1/1] test.PageAllocator resize... FAIL (TestUnexpectedResult)
/snap/zig/7908/lib/std/testing.zig:515:14: 0x20a5b7 in expect (test)
    if (!ok) return error.TestUnexpectedResult;
             ^
/tmp/test.zig:6:3: 0x20a733 in test.PageAllocator resize (test)
  try std.testing.expect(page_allocator.resize(buf, 100 + std.mem.page_size) == true);
  ^
0 passed; 0 skipped; 1 failed.
1 Like
4

Might be worth noting these parts of the doc comment for resize:

/// Attempt to expand or shrink memory in place. [...]
/// 
/// A result of `true` indicates the resize was successful and the
/// allocation now has the same address but a size of `new_len`. `false`
/// indicates the resize could not be completed without moving the
/// allocation to a different address.

If you’re just looking to increase the size of the allocation, realloc might be what you’re looking for instead. If you think the resize implementation could guarantee the re-map without moving the address in your test case, then it’d be worth filing an issue for sure.

1 Like
5

Might be worth noting these parts of the doc comment for resize:

Yes I saw that but as:

  • mremap seems to be able to fail if resize is impossible (from the man page: By default, if there is not sufficient space to expand a mapping at its current location, then mremap() fails)
  • mremap is suggested in the comment
    I though that this requirement was “enforceable” with mremap. I might try to push a PR although I don’t know what would be the requirement for it to be accepted…

If you’re just looking to increase the size of the allocation, realloc might be what you’re looking for instead.

This is where it gets a little fuzzy for me because realloc don’t seem to be part of the Vtable interface to Allocator so PageAllocator could not provide that function. Yet realloc is part of the public interface of the Allocator struct. For example, there are a couple of realloc test in general_purpose_allocator.zig but no implementation, so I assumed it is testing the function in Allocator.zig.

6

Just to add to this - you can see a resize being done here in the ArrayList implementation if the resize fails, from ensureTotalCapacityPrecise:


            const old_memory = self.allocatedSlice();
            if (self.allocator.resize(old_memory, new_capacity)) {
                self.capacity = new_capacity;
            } else {
                const new_memory = try self.allocator.alignedAlloc(T, alignment, new_capacity);
                @memcpy(new_memory[0..self.items.len], self.items);
                self.allocator.free(old_memory);
                self.items.ptr = new_memory.ptr;
                self.capacity = new_memory.len;
            }
1 Like
7

Not quite sure what you mean by this. std.heap.page_allocator is a (slightly special) Allocator instance, so it has access to the std.mem.Allocator functions.

This works, for example:

const std = @import("std");

test "PageAllocator realloc" {
    const page_allocator = std.heap.page_allocator;
    var buf = try page_allocator.alloc(i32, 100);
    var new_buf = try page_allocator.realloc(buf, 100 + std.mem.page_size);
    std.debug.print("{*} -> {*}\n", .{ buf, new_buf });
}

See

8

I think what may be confusing here is the fact that the entire file in the Allocator.zig is actually being used like a struct - the virtual table is only a part of it. So because of that, it may look like the table itself is the only object in that file, but the whole file is the actual object though.

zig/lib/std/mem/Allocator.zig at master · ziglang/zig · GitHub

At the top, you can see that they declare:

const Allocator = @This();