Windows file name handling / large stack use

Brainstorming
1

I was investigating why my coroutines need 2MB of stack space to run simple file operation on Windows and realized that it’s due to the large stack allocated buffers in std.os.windows. For now I’ll accept it, but I’d really like to fix it eventually, I want to be able to run coroutines with much smaller stacks.

I had one idea, how I could deal with that if I really wanted to. I COULD write stackAlloc(T, n) using inline asm hack, and reimplement all these functions to only allocate as much stack as they really need. The conversions always happens locally, I pass u8 slice as file name, it converts to UTF-16, and passes to Windows function call. There is no risk of leaking these pointers elsewhere.

I think this is the only case where stack allocator would make sense, the file names are bounded. It’s also possible that the Windows C library uses large stack buffers like this and then I have no option, but I was still curious because even the default Windows Fiber context only uses 1MB stack by default and I get stack overflow if I use that size. While on Linux, I could use 64KB stack and it’s just fine.

I don’t really suggest adding stackAlloc / alloca to the language, but I wonder if people have different ideas how to deal with this. Or should we just accept the stack use?

2

Tbh, for a thin Windows API wrapper I would expect that the caller needs to take care of any UTF-8/16 conversion, and then provide a separate UTF-8/16/32 conversion helper which takes an allocator somewhere else in the stdlib.

Also just for perspective, Emscripten’s default stack size is 64 Kilo-bytes :slight_smile:

PS: interestingly, std.os.windows doesn’t seem to agree whether strings are UTF-8 or UTF-16, for instance OpenFile() takes an UTF-16 string:

…while GetFileAttributes() takes an UTF-8 string and does the conversion internally:

…I hope that OpenFile() is the new way and the other functions just haven’t been updated yet…

3

In terms of stack usage, there is definitely room for improvement. I will get this merged soon which will help in certain scenarios.

Some notes:

  • Mentions of UTF-8/UTF-16 should be WTF-8/WTF-16.
  • std.os.windows.GetFileAttributes is just a wrapper around std.os.windows.GetFileAttributesW which deals in WTF-16. I agree, though, that the naming should be standardized, and that the only std.os.windows functions that deal with WTF-8 should be the conversion functions (sliceToPrefixedFileW, etc).
1 Like
4

There is a simple solution to this problem (encoding conversion buffers), if it is acceptable to have a recent Windows system (latest Windows 10 and later, Windows Server 2019 and later), you can directly call the -A version of the functions (instead of the -W version) and set the process code page to utf-8.
See: Use UTF-8 code pages in Windows apps - Windows apps | Microsoft Learn

5

This strategy doesn’t always work. There are some APIs where the A versions are hamstrung versions of the W versions, with fewer capabilities. (WinSock) They are rare, but they exist. It’s possible that this is something that will eventually go away, but the full utf8-ization of Windows is a WIP.

This requires adding a manifest to your EXE, which would be nice if there was nicer toolchain integration for.

it would be interesting to do some testing to see if the Windows A conversion thunks enforce utf-8 (where invalid sequences error or substitute the Object Replacement Character), or do they allow wtf-8? The doc page doesn’t clarify, but suggest you to enforce utf-8 when they provide that as the recommended way to convert using MultiByteToWideChar.

2 Likes
6

As far as I know, there are a few reasons why that’s not a straightforward improvement:

  • I’ve seen it mentioned that the conversion still happens, it’d just happen within the Windows code instead of in Zig code
  • There still may be APIs that don’t behave fully correctly; the Use Unicode UTF-8 for worldwide language support option is still marked as Beta AFAIK
  • The recommended method of enabling UTF-8 support via the A suffixed functions is unfortunate; it requires embedding a manifest which introduces some unknowns (which may not be real problems, I just don’t know the answers):
    1. If the user wants to provide a manifest, then something needs to be done to reconcile the generated activeCodePage manifest and the user’s manifest. Merging manifest files would need the involvement of an xml parser (right now Zig only allows 1 manifest file)
    2. The artifact being built may not have control over the process that loads it, e.g. I’m not sure how a manifest in a .dll interacts with activeCodePage. I’m guessing that using the A functions from a .dll is not guaranteed to behave as expected, though.

Are you aware of the win32_manifest field of ExecutableOptions/LibraryOptions?

const exe = b.addExecutable(.{
    .name = "foo",
    .root_module = b.createModule(.{
        // ...
    }),
    .win32_manifest = b.path("res/foo.manifest"),
});

(or you can just pass a .manifest file to build-exe/build-lib and it’ll work, see here for the initial PR that added manifest support)

4 Likes
7

You can also use a normal allocator for these kinds of small, localized allocations.
In my project I’m using a threadlocal allocator for all local allocations.
It’s implemented similar to the stack, with a small extension to allow using it for more data structures, like lists. And it also has a fallback to the global allocator, which makes it even more resilient than normal stack allocation.

8

Yes, that is true. Windows converts -A to -W calls in user space.

That global flag in windows broke all non unicode applications worldwide. But the manifest setting is fine.

You can also have the manifest externally with .exe as .exe.manifest

You are also correct on this. A dll cannot control the conversion, it is up to the process exe manifest to decide the code page.

No, thank you for mentioning it.

2 Likes
9

Windows Version 1903 (May 2019 Update)

…might be a bit too recent (considering that we only ended Windows XP support in our PC game in 2019 because there was still a single-digit percentage Windows XP users at that time - most likely in China)

10

Regarding this builtin UTF-8 support in Windows via the process codepage, I also looked into this a couple of years ago and came to the conclusion that it’s not worth it (what I do though is setting the console codepage to UTF-8 programmatically so that log messages to the console via the C stdlib string output functions look alright - I don’t think there’s a programmatic way to set the ‘process codepage’ though - only via a manifest.xml).

Instead in my C code I convert between UTF-8/16 (ok actually WTF-8/16) at the API boundary (e.g. right before or after a Window ‘W’ call), and I also call the W functions explicitly (e.g. not via the macro which selects between the A and W version.

In Zig I would expect if there are Windows API wrapper functions which take WTF-8 strings that they also take an allocator for the temporary string conversion allocation. But tbh - why bother, just directly expose the 16-bit string functions…