Hello everyone, I spent some time making this writeup:
https://andrewkelley.me/post/zig-new-cli-progress-bar-explained.html
Hope you enjoy!
By statically pre-allocating 200 nodes, the API is made infallible and non-heap-allocating
Statically allocated arrays are a C insight we should never have forgotten. 80%¹ of the time all you need is a 1000 of something and that’s it. Add an error condition if you need more than those 1000 if it helps you sleep at night and you’ll be fine.
Anyway, thanks for that write up!
Âą completely not backed up.
I tried on my old PC with an Intel haswell cpu, and the flickering is a bit annoying.
Is it possible to auto-detect the refresh rate based on the cpu or add a command line flag?
I used the latest zig release and urxvt terminal.
Thanks.
Can you share more details about this flickering? I expect to see no flickering regardless of CPU.
This is the asciinema link. Unfortunately the web player does not reproduce the problem:
asciinema
This is the link of the recording done with ffmpeg:
youtube
Thanks.
This would be awesome for Linux distro package managers, letting you actually see why if I’m installing package X, packages B, H, and M are being installed. If I recall correctly, right now, you just get informed they need to be installed, but you don’t see what depends on what. Makes me wonder if a mode in which the tree just keeps expanding and doesn’t redraw would be useful to see the whole tree when the process finishes.
1 Like
urxvt is listed as unknown in the gist you’ve linked to about synchronized output. It looks like it sees the screen after the clear to end of screen sequence sometimes, creating the flicker.
Instead of erasing everything and then writing, you could issue "\x1b[K" at the end of each line of text, then a newline.
Terminals might see a partial render, but this should prevent flickering on any platform, since the text is never entirely gone. Makes sense to use the synchronized output sequence for the terminals which support it, but the above should give better results for ones which don’t.
1 Like
The ZIG_PROGRESS seems kind of limiting:
-
it unnecessarily confined the process to be on the same host. I can’t see the actual progress API, but hopefully it just takes in a an arbitrary socket to allow for the parent and children to be on different hosts.
-
It limits the process so just a single progress bar it reports
If the underlying takes in a fd and doesn’t have that hardcodes, then above doesn’t matter.
Why use ZIG_PROGRESS to send an fd to children: either open the pipe prefork and save its fd off, send the fd across a control pipe, use a named pipe, and prob others. I’m not sure what the benefit is of using env vars (storing the fd in a global just seems so much easier).
I can already imagine the used cases for the pipe being named pipe or socket. GMake and related tooling have a job server originally to prevent recursive makes from all spawning 20 processes, but now a number of tools like parallel can all be given a pipe name and use the same job pool. I can see the same for the zig infra
I tried alacritty and it works correctly.
Thanks.
2 Likes
That’s what it does, it opens the pipe prefork. The child process then needs to know which of its file descriptors is the pipe.
The progress pipe is the control pipe. If you have a control pipe, then how do you communicate to the child process which pipe is the control pipe? A meta control pipe? Now we need a meta meta control pipe. I don’t understand this suggestion at all.
And then communicate that name how?
store it in an int (that’s all your really doing when you set the environment variable anyways, just it is using a string with a specific format in a specific region of memory - they are all in the process and kernel plays no part in it - but now you have to parse it back out the environement block).
You might be able to see the env var in /proc/*/environ that’s a nice plus, but sadly that block can be moved by the act of using setenv and then you just get garbage from the proc file.
https://www.cse.cuhk.edu.hk/~ericlo/teaching/os/lab/6-IPC1/pipe-fork.html
shows the usual way.
It seems like you’re assuming child processes can access variables from their parents? That’s not how it works. You’re linking to an example that teaches about fork() and does not have a call to execve()… I’m confused why you’re speaking authoritatively about this topic while making nonsensical suggestions.
2 Likes
Do you already have an idea on how to make ZIG_PROGRESS work on Windows? Off the top of my head, I imagine we’d use a pipe like we use for the std streams in std.process.Child. In that case we use a standardized format for the pipe name and namespace it to the child process by including the child PID in the name, i.e.
\\.\pipe\zig-progress-${CHILD_PID}
In this case we’d have no need for an fd number as the pipe name would already be namespaced to the process and you can only have 1. We could still signal to the child process that there is a zig progress pipe available via an environment variable, however, I believe this would be unnecessary since the child process could just check if the progress pipe exists with the expected name.
P.S. thanks for writing all this up, it was a fun read
I can’t see code so I ddon’t know whats parts of the progress meter is in your contrrol or the end goals. At the base there is always some convention (eg, in a particular environmet var or on a well known fd), I usually just have all children have the same fd numbers.
Parent has 3, 4, and 5 readable fds, but each child has a single writable fd 3 (or pick 99 or whatever number you want and dup2 it into place).
I’ve seen complicated schemes to use named pipes (based on pids), memfds than get mmap’ed post exec, etc but they just seem like a more complex way to do the same thing.
That’s a neat idea. Does the parent have access to the child pid before the child spawns though?
Here is Jacob’s WIP PR. His strategy is to create the pipe just like a stdout or stderr pipe, tell NtCreateUserProcess to include it as one of the inherited handles, and then rely on the ZIG_PROGRESS environment variable to communicate the handle’s numerical value, just like the posix version.
Does the parent have access to the child pid before the child spawns though?
Oh I am remembering this slightly wrong. We actually use the parent PID to namespace the std pipe names, not the child PID. Looks like we also use a global pipe name counter to prevent conflicts from multiple child processes.
That being said, on Windows you actually can create a child process in a suspended state by passing the CREATE_SUSPENDED flag to CreateProcess (here’s an example). So we could create the child process in a suspended state, then create the named pipe with the child PID, then call ResumeThread to allow the child process to actually start.
I’ll check out Jacob’s PR, looks interesting right off the bat but I’m not immediately understanding it 
2 Likes
Visualisation can be a great tool to detect what is going on “in between” and detect bugs even if the end-result is ok.
I used to work on Z-buffer rendering to memory, dumping the memory to (GIF) file and then view the result (256 colours out of 16M was state of the art at that time).
During a visit to Dublin to adapt the GIF show program we had, to the Artist XJS graphics adapter card, I learned that the video memory of that card could be mapped into the extended memory of the (then top of the line) Intel 386. Using that mapped in memory as the rendering buffer, allowed us to find some bugs in the processing as well, that did not show up in the end-result because they were covered up.
The key insight I had here is that, since the end result must be displayed on a terminal screen, there is a reasonably small upper bound on how much memory is required, beyond which point the extra memory couldn’t be utilized because it wouldn’t fit on the terminal screen anyway.
This is a very general insight! For a lot of computer things, the end consumer is a human, and humans can process only so much information. So, it often (but not always) makes sense to try to show less information, and instead increase relevance and reduce latency. And that usually requires pushing the max limit into the system, so that no work is spend generating useless data, as opposed to the default of throwing away 90% of the information that doesn’t fit on the screen as the last step of computation.
4 Likes
Is it correct to say that now std.Progress is one of the most complex piece in the stdlib?
AFAIUI, to show “less” in this case, one needs to pass --color off to zig now, even though that won’t prevent the work from being done.