Possible problems with fs.File.getOrEnableAnsiEscapeSupport

Recently I noted that in fs.File there is a new method getOrEnableAnsiEscapeSupport.

This method actually force the VT Sequences processing support to be enabled for the legacy console.

The problem is that in windows: add support for native virtual terminal by perillo · Pull Request #15206 · ziglang/zig · GitHub @squeek502 was worried that this will affects all processes run in the console session, not just the current process.

I got a confirmation in proposal: x/term: add detection of VT sequences processing support · Issue #73415 · golang/go · GitHub.

I think that this problem should be documented in the getOrEnableAnsiEscapeSupport function.

Should I create a new issue?

Thanks.

If you confirm it to be a problem, then sure. I didn’t read the whole discussion on GitHib, but it appears that this concern is walked back in next reply.

This is similar to #7600 (see also #12400 and #14411) in that it affects all processes run in the console session, not just the current process

I was wrong about this, apologies for not checking my assumptions. See #20172 for a revivial of the spirit of this PR.

I personally wouldn’t be creating additional noise in the issue tracker without a minimal example of it actually behaving incorrectly, or based only on assumption, but I can understand there might be exceptions to this.

Isn’t this only needed for the old Windows command prompt. AFAIK it’s enabled by default in Windows Terminal.

Since it is not clear if this it is really unsafe, I will not open an issues.

This will require some testing on a Windows machine.
I have one, but it runs on windows 11

Thanks.

You can test this.

Test program
const std = @import("std");

pub fn main() !void {
    var arena_state = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena_state.deinit();
    const arena = arena_state.allocator();

    const args = try std.process.argsAlloc(arena);

    if (args.len <= 1) {
        return error.MissingArguments;
    }

    const stderr = std.io.getStdErr();
    if (std.mem.eql(u8, args[1], "get")) {
        std.debug.print("supportsAnsiEscapeCodes: {}\n", .{stderr.supportsAnsiEscapeCodes()});
    } else if (std.mem.eql(u8, args[1], "getorset")) {
        std.debug.print("getOrEnableAnsiEscapeSupport: {}\n", .{stderr.getOrEnableAnsiEscapeSupport()});
    } else if (std.mem.eql(u8, args[1], "unset")) {
        var original_console_mode: std.os.windows.DWORD = 0;
        if (std.os.windows.kernel32.GetConsoleMode(stderr.handle, &original_console_mode) != 0) {
            const console_mode = original_console_mode ^ std.os.windows.ENABLE_VIRTUAL_TERMINAL_PROCESSING;
            if (std.os.windows.kernel32.SetConsoleMode(stderr.handle, console_mode) != 0) {
                std.debug.print("unset ENABLE_VIRTUAL_TERMINAL_PROCESSING\n", .{});
            }
        }
        std.debug.print("supportsAnsiEscapeCodes: {}\n", .{stderr.supportsAnsiEscapeCodes()});
    }

    const escape_code_tty = std.io.tty.Config{ .escape_codes = {} };
    try escape_code_tty.setColor(stderr.writer(), .green);
    std.debug.print("hello\n", .{});
    try escape_code_tty.setColor(stderr.writer(), .reset);
}

In Windows Terminal (where ENABLE_VIRTUAL_TERMINAL_PROCESSING is enabled by default):

> vt.exe get
supportsAnsiEscapeCodes: true
hello

> vt.exe unset
unset ENABLE_VIRTUAL_TERMINAL_PROCESSING
supportsAnsiEscapeCodes: false
←[32mhello←[0m

> vt.exe get
supportsAnsiEscapeCodes: true
hello

In cmd.exe (where ENABLE_VIRTUAL_TERMINAL_PROCESSING is disabled by default, but can be enabled):

> vt.exe get
supportsAnsiEscapeCodes: false
←[32mhello←[0m

> vt.exe getorset
getOrEnableAnsiEscapeSupport: true
hello

> vt.exe get
supportsAnsiEscapeCodes: false
←[32mhello←[0m

That is, the setting does not persist between subsequent processes. However, note the caveat mentioned here:

A caveat here is that child processes can affect the console mode of parent processes, see #16526 (comment) for an example of how this can cause problems. So this may not actually be fully safe to do, as spawning Zig as a child process can end up messing with the console mode of other processes unexpectedly.


This is different than how the code page setting works.

Code page test program
const std = @import("std");

pub fn main() !void {
    std.debug.print("→ code page at startup: {}\n", .{std.os.windows.kernel32.GetConsoleOutputCP()});

    std.debug.print("→ setting code page to 65001 (UTF-8)...\n", .{});
    std.debug.assert(std.os.windows.kernel32.SetConsoleOutputCP(65001) != 0);
    std.debug.print("→ done\n", .{});

    std.debug.print("→ code page at exit: {}\n", .{std.os.windows.kernel32.GetConsoleOutputCP()});
}
> codepage.exe
→ code page at startup: 437
→ setting code page to 65001 (UTF-8)...
→ done
→ code page at exit: 65001

> codepage.exe
→ code page at startup: 65001
→ setting code page to 65001 (UTF-8)...
→ done
→ code page at exit: 65001

(same thing happens in both Windows Terminal and cmd.exe)

(weird side note: running chcp still reports Active code page: 437 even though GetConsoleOutputCP returns 65001; EDIT: chcp seems to look at the input code page, calling SetConsoleCP changes what chcp reports)

FWIW, you can still use the old Command Prompt. There’s a setting somewhere to configure whether Command Prompt or Windows Terminal gets used, though I don’t remember if it’s in Windows settings or in Windows Terminal settings.

That aside, having worked extensively with the Win32 console APIs previously, I can say that Ryan is right; Windows console state is not scoped to a single process. So I would agree that this should be documented.

I wonder if this is a feature of the Win32 console infrastructure, the shell, the terminal, or some combination of these.

The fact that child processes can affect the console mode of the parent process suggests to me that it might not be a feature of the console infrastructure.

1 Like

I don’t have an answer to that, but maybe this workaround mentioned here can provide a clue?

Another possible workaround is use to cmd /c when spawning git, since it seems like the console mode is restored by the terminal in that case.

But that might be explained by cmd /c creating a separate screen buffer.