Why does iterating over command line arguments require an allocator on windows?

kj4tmp · February 13, 2025, 4:34am

From std.process we can see that iterating over the command line arguments on windows requires an allocator, while on linux, it does not.

Why does iterating over command line arguments on windows require an allocator?

github.com

ziglang/zig/blob/5b9b5e45cb710ddaad1a97813d1619755eb35a98/lib/std/process.zig#L1168


      
          const InnerType = switch (native_os) {
              .windows => ArgIteratorWindows,
              .wasi => if (builtin.link_libc) ArgIteratorPosix else ArgIteratorWasi,
              else => ArgIteratorPosix,
          };
          
          inner: InnerType,
          
          /// Initialize the args iterator. Consider using initWithAllocator() instead
          /// for cross-platform compatibility.
          pub fn init() ArgIterator {
              if (native_os == .wasi) {
                  @compileError("In WASI, use initWithAllocator instead.");
              }
              if (native_os == .windows) {
                  @compileError("In Windows, use initWithAllocator instead.");
              }
          
              return ArgIterator{ .inner = InnerType.init() };
          }

AndrewKraevskii · February 13, 2025, 5:33am

This is what I found after reading standart library:

On Linux, arguments are stored in a known location within the loaded binary, already processed and ready for use as-is, so the iterator can simply return a slice of them.
On Windows, however, the arguments are not pre-processed, so the iterator must handle this task. Since it needs to modify the arguments, it must copy them into its own buffer. So allocator is required to obtain said buffer.

Here we can see windows iterator handling escaping, unescaping and skiping whitespace.

github.com

ziglang/zig/blob/5b9b5e45cb710ddaad1a97813d1619755eb35a98/lib/std/process.zig#L1073-L1131


      
          pub fn next(self: *Self) ?[:0]const u8 {
              if (!self.skipWhitespace()) {
                  return null;
              }
          
              var backslash_count: usize = 0;
              var in_quote = false;
              while (true) : (self.index += 1) {
                  const character = if (self.index != self.cmd_line.len) self.cmd_line[self.index] else 0;
                  switch (character) {
                      0 => {
                          self.emitBackslashes(backslash_count);
                          self.buffer[self.end] = 0;
                          const token = self.buffer[self.start..self.end :0];
                          self.end += 1;
                          self.start = self.end;
                          return token;
                      },
                      '"', '\'' => {
                          if (!options.single_quotes and character == '\'') {

This file has been truncated. show original

While posix iterator just does nothing

pub fn next(self: *ArgIteratorPosix) ?[:0]const u8 {
    if (self.index == self.count) return null;

    const s = std.os.argv[self.index];
    self.index += 1;
    return mem.sliceTo(s, 0);
}

kristoff · February 13, 2025, 9:50am

There’s also a more basic problem: on windows you get wtf16 encoded strings which are a pain in the ass to work with, so this interface transcodes them to utf8.

github.com

ziglang/zig/blob/5b9b5e45cb710ddaad1a97813d1619755eb35a98/lib/std/process.zig#L712-L734


      
          /// `cmd_line_w` *must* be a WTF16-LE-encoded string.
          ///
          /// The iterator stores and uses `cmd_line_w`, so its memory must be valid for
          /// at least as long as the returned ArgIteratorWindows.
          pub fn init(allocator: Allocator, cmd_line_w: []const u16) InitError!ArgIteratorWindows {
              const wtf8_len = unicode.calcWtf8Len(cmd_line_w);
          
              // This buffer must be large enough to contain contiguous NUL-terminated slices
              // of each argument.
              // - During parsing, the length of a parsed argument will always be equal to
              //   to less than its unparsed length
              // - The first argument needs one extra byte of space allocated for its NUL
              //   terminator, but for each subsequent argument the necessary whitespace
              //   between arguments guarantees room for their NUL terminator(s).
              const buffer = try allocator.alloc(u8, wtf8_len + 1);
              errdefer allocator.free(buffer);
          
              return .{
                  .allocator = allocator,
                  .cmd_line = cmd_line_w,

This file has been truncated. show original

floooh · February 13, 2025, 10:35am

FWIW I just use argsWithAllocator() on all platforms, I actually thought the ‘other’ args() function is deprecated…

Would be nice if the Zig stdlib exposed a simple higher-level arg-parser, e.g. give it a array of accepted args (with long and short name, type, description and optional default value) and return a struct with the ‘resolved’ args (I know that such libs exist as 3rd party, but it’s such a common thing for command line tools that stdlib support would be nice). Could also use Zig comptime to build that resolved-args struct type.

squeek502 · February 14, 2025, 12:59am

Note that you linked the next implementation of ArgIteratorGeneral, which is currently only used for parsing response files. Here’s ArgIteratorWindows, which (as noted earlier) takes a single WTF-16 string as input:

github.com

ziglang/zig/blob/5b9b5e45cb710ddaad1a97813d1619755eb35a98/lib/std/process.zig#L682-L950


      
          /// Iterator that implements the Windows command-line parsing algorithm.
          /// The implementation is intended to be compatible with the post-2008 C runtime,
          /// but is *not* intended to be compatible with `CommandLineToArgvW` since
          /// `CommandLineToArgvW` uses the pre-2008 parsing rules.
          ///
          /// This iterator faithfully implements the parsing behavior observed from the C runtime with
          /// one exception: if the command-line string is empty, the iterator will immediately complete
          /// without returning any arguments (whereas the C runtime will return a single argument
          /// representing the name of the current executable).
          ///
          /// The essential parts of the algorithm are described in Microsoft's documentation:
          ///
          /// - https://learn.microsoft.com/en-us/cpp/cpp/main-function-command-line-args?view=msvc-170#parsing-c-command-line-arguments
          ///
          /// David Deley explains some additional undocumented quirks in great detail:
          ///
          /// - https://daviddeley.com/autohotkey/parameters/parameters.htm#WINCRULES
          pub const ArgIteratorWindows = struct {
              allocator: Allocator,
              /// Encoded as WTF-16 LE.

This file has been truncated. show original