I have seen software fail too often to ever trust that it’s correct, so I will never pass up an opportunity to test it more. Always ReleaseSafe for me ![]()
Also relevant is, how many of the bugs found my LLMs were security vulnerabilities?
Unfortunately not as obvious as we all wish it were, the Cyber Resilience Act introduces some requirements for open source code.
My main grip with release safe is that arithmeric code gets pretty bad codegen, and counterintuitive checks are put at the wrong place. But I can turn off checks locally.
Otherwise I do like the idea that my asserts are still asserted in prod, but we haven’t seen any so far in a public release
I don’t think we have opposite view points! I mostly agree with you.
I think the recent vulnerabilities have me spooked and wondering if I can get to a point where I have enough confidence about my software to remove what I consider the last guardrails. Maybe just an emotional response though … I keep trying to remind myself this stuff isn’t magic, you can understand it, but I’m not there yet ![]()
I would be more willing to use ReleaseSafe instead of ReleaseFast in a real environment if ReleaseSafe gives a catchable error instead of crashing when issues other than memory corruption occur. Crashes in production need to be handled very carefully, as they could lead to unacceptable consequences.
…it’s the only right thing to do tbh… every ‘predictable’ and recoverable error should be handled via regular error handling, everything else needs to crash so that the problem isn’t exploitable. And in such ‘unexpected’ situations (like an out-of-range write) the program will most likely be in a state where it cannot safely continue anyway, since critical data may have been corrupted by the out-of-range write.
You can probably define a custom panic handler which (maybe?) is able to save the currently loaded data so that it might be recovered on a restart (and also get some telemetry about the crash out).
…also for server-side stuff you should have a solution which monitors running processes and restarts them when they “disappear” for whatever reason (basically Erlangs “let it crash” philosophy).
I aspire to this but I don’t have it set up yet - can you suggest a good Zig implementation I can use as a model?
Why reinvent the wheel?
On Linux, there’s systemd for example.
In addition to not knowing how to use that particular wheel, I’m hoping for example monitoring code that I could use on macOS, Windows, OpenBSD and occasionally Linux. You make a good point though and I’m open to non-Zig but widely portable suggestions though.
You might look into what the Zig test runner does, since it has to handle the test program crashing.
I developed a mature process management program in C++ for Windows, and for Linux an Windows, I wrote a similar tool in Python (think of it as a poor man’s systemd without bloat). Both are closed source, unfortunately. I’m a Zig noob more or less, so I wouldn’t dare doing this myself in Zig yet.
There is the zig-process library on GH, which looks reasonable, according to its description. It is very young, so not yet “battle-tested”, but it may be worth a try.
Thanks to you and @alanza both, I’ll look at the Zig test runner and the zig-process and see what I can easily borrow! Battle-tested isn’t as huge a requirement since I’ll be reimplementing it to fit anyway, but just in case I’ll start with the Zig test runner.
Something I keep having to remind people at work is, safety checks do not remove safety bugs. They just make safety bugs noisy. You don’t want a safety bug to happen when something is released regardless of whether safety checks are there or not.
Sure you might restart your processes on a safety failure, and that will stop an information leak or an RCE, but now you have a denial of service exploit.
The power of safety checks are in development when coupled with a very good test suite. That doesn’t mean 100% line coverage. That’s far too easy. It has to go far beyond that. Your tests need to try to break your code. Try to make those checks fire. It’s a different mindset to most testing you see out there.
If you have that, switching the checks off is fine because you have the confidence that they are redundant code.
I wouldn’t solve this with a custom Zig tool, there’s plenty of standard solutions around that monitor and restart processes, like supervisord or monit (or even systemd).
I wasn’t expecting these sorts of answers but I really appreciate them. Let me explain:
I was imagining something more like OpenSSH, which *is* crash-only with isolated processes, but *also* is a single integrated system factored into isolated privilege-dropped processes. That ‘privilege separation’ creates some of the safest networking C code that I know of. I know I can use things like directly imsg_init(3) - OpenBSD manual pages (portable example openiked-portable/compat/imsg.c at f0da8188ae8b1778dbdd215f7bc30bd8f166aa5e · openiked/openiked-portable · GitHub ) but I’d love to see a portable native Zig take on this.
So when I hear about running another process “crash-only”, this is what I was picturing. Maybe because I don’t know Erlang, only this one idea from it. Also, I was totally unaware of most of these tools, coming from a mostly Windows background, so thank you all, I have more reading to do!
(Disclaimer: I don’t necessarily want to call myself the expert on our security discipline! But this is what I got from asking the team myself.)
We do a lot of things in Ghostty to validate the integrity of the project, particularly libghostty; we do the standard things like fuzz tests and Valgrind, but honestly I think more importantly is a general discipline around testing and integrity verification - we have a ton of tests (approx. 4.2k from my last count), which run in debug mode of course, and we will also add asserts and even higher-level integrity checks (such as verifyIntegrity) that only run in safe builds.
So my answer would be probably be that having a good culture around software quality goes a long way. Having a diverse user base probably helps too.
However, I personally credit Zig’s front-and-center safety features with helping this: the documentation makes several references to how illegal behavior is handled, and safety checks need to be explicitly turned off. So probably copious amounts of column A and B. ![]()
BTW: Mitchell did just recently add a parallel test runner to Ghostty, forked off of the upstream test runner. It slashed our run times down a lot, if anyone’s interested.
Oh also, another good reference for general quality culture: Andrew’s State of the Tagged Union Address where he talks about total bug elimination: at 29:40.
Until I find a way to unify Debug for development, ReleaseSafe for shipping/debugging further I think I cannot consider myself skilled enough to ship things with ReleaseSmall/ReleaseFast.
(post deleted by author)