I’ve always thought that disabling the safety checks inserted by the compiler is a bit crazy.
In light of recent slew of xyz2shell vulnerabilities (nginx today), I figured I’d actually ask the community to figure out how people think about this tradeoff:
If you build your zig programs with safety checks disabled, how do you sleep at night friends?
Like it feels like I would want the safety checks on all the time, especially in these times.
The sky won’t fall when your Pacman clone has a memory corruption issue.
For code that needs to run on servers with access to sensitive user data, or generally code that handles secrets that’s a different issue of course.
But even with memory corruption in place, there’s other ‘onion layers’ before it can be exploited (sandboxes like OS processes, VMs like WASM, network firewalls etc…), and in an ideal world, those sandboxes need to be watertight and prevent even the worst bug-riddled code from running amok outside the sandbox (WASM is probably closest to that promise).
For actually safety-critical code, ReleaseSafe isn’t enough anyway since it basically only adds range checks (and it avoids that failed asserts turn into undefined behaviour, which I still think is a stupid^H^H^H erm, “sub-optimal” decision) - and btw: Rust also removes runtime range checks in release mode (misinformation alert!) . You’d need to move to something like validated MISRA code or a radical solution like Fil-C (at a pretty hefty performance hit) to actually “sleep at night”
This is a bit of a dangerously ambiguous phrasing I’d say. I am not sure if you are talking about numeric integer range checks (overflows) or array/slice index range checks (bounds checks). I assume it’s the latter, as it is more important to avoid exploitation.
There are two things in play here:
Optimizing compilers, including LLVM, can and do eliminate provably redundant bounds checks for slice accesses. Eg, in code like for (0..xs.len) |index| use(xs[index]), there won’t actually be a bounds check, if the compiler is doing even just the basic optimizations.
In fast/small modes, Zig removes the checks even if it can’t prove they are redundant. That would be code like for (0..100) |index| use(xs[index]).
Rust will never do the second thing unless you do all of
compile in release mode
write unsafe {
use designated unchecked API like slice::get_unchecked.
(for integer range checks, yes, Zig’s behavior is safer than Rust’s behavior, though by itself integer overflow isn’t exploitable. Integer overflows in C are usually problematic, because they often break programmer-instead bounds check. Eg,if (base + user_input < size) can pass even if user_input > size)
I am also wondering this. There’s a specific empirical question here:
As far as I understand, ghostty and libghostty are distributed in ReleaseFast. How come we don’t have a string of high-profile vulnerabilities in that (widely deployed) software, while C stuff is popped pretty regularly?
I still don’ know, but the best guess I’ve heard is that having slices as a language construct, even without bounds checks, goes a long way toward closing index-out-of-bounds holes, which in turns goes a long way of making software non-exploitable.
I’d love to read a writetup of investigation of ghostty security posture, with respect to memory safety vulnerabilities.
I almost agree with you, but it’s not always easy to know if something can be a vector for an attack. I find I have to switch to thinking “creatively”. So for example: if a hypothetical pacman game can say, load a progress file, then you all of a sudden have to think about someone engineering a malicious progress file and sending it to an unsuspecting fellow pacman player.
For actually safety-critical code, ReleaseSafe isn’t enough anyway since it basically only adds range checks
You might be right. I guess i posted within the context of the recent vulnerabilities being about taking advantage of a use-after-free/out-of-bound-writes to get remote code execution Seems like ReleaseSafe (assuming the SafeAllocator is used) would help catch these types of things by panicking.
For my IRC server/client I do try to assert everything, and handle all errors, but I dont really pay attention to optimization options besides knowing that they may or may not cause me issues in crypto code.
If each build setting passes the test suit, and there was exploitable bug, then it was issue with test suit, and not with some build settings. At that point, logging would be nice to know that something is or was happening, but then what optimization has to do with logging?
Ooops, sorry for the misinformation… I must have mixed that up with overflow checks (or rather assumed that both ‘optimizations’ fall into the same category).
I edited my post to make the wrong part clear. Thanks!
This is actually what I believe is the belief behind letting you choose the optimisation level at scope level. As it usually goes, I would recommend not optimising before benchmarking (partly because it’s really fun seeing number go down), and benchmarking here gives us the ability to go and be like, oh, right here the compiler seems to output code that is a bit slower than it’s ReleaseFast counterpart, but an issue here is not critical to me (or perhaps it is not even possible due to an not described invariant), I can go ahead and turn off pesky safety here and here only.
That’s the common case situation. A lot of projects which use ReleaseFast or Small are ones that really need it. If you are writing a chess engine you couldn’t really care about safety in a competition - sure, in development you might want to consider it in order to catch UB, but not on release.
That’s a good guess, but for Ghostty/libghostty and Zig in general I think it’s too early to conclude anything. Even though libghostty is now embedded in several big products, this all only happened fairly recently, and it’s a much smaller attack target than “is installed by default in every Linux distro or on each AWS or Google cloud server”.
I hope you are at least aware of Zig’s assert behaviour in ReleaseFast and ReleaseSmall mode (the compiler basically ‘injects’ undefined behaviour in order to turn the assert into an optimization hint, e.g. as if a C assert() would turn into __builtin_unreachable() in release mode):
E.g. quote from the Zig docs:
In fast and small mode, the optimizer uses the assumption that unreachable code will never be hit to perform optimizations.
That means that if your tests in debug or safe mode didn’t catch all potential assert inputs, any assert that would be triggered in fast- or small-mode is now active undefined behaviour where real weird things might happen instead of a simple panic or no-op
(of course if the assert is simply removed instead of being used as “optimization hint” this would also open up the code to other problems, but probably not quite as on-the-nose as injected UB).
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
On a more serious note though: Yeah, maybe you can catch a few vulnerabilities, but the safest program cannot prevent the biggest security vulnerability of them all: The User.
Also one thing to consider is what is the cost of the possible vulnerability vs the cost of ReleaseSafe. Maybe collectively the energy and time savings of your program outweighs the cost of a few people that get hacked due to a vulnerability. Or maybe ReleaseFast makes your program less likely to crash, and there are applications where a crash could cost lives.
The useful dividing line is the trust boundary, not the whole binary. I’d keep network-facing and parser code in ReleaseSafe, then use scoped @setRuntimeSafety(false) only for inner loops operating on already validated data. ReleaseFast changes the failure mode rather than making code secure, and fuzzing in Debug/ReleaseSafe still matters because checks only catch bugs they actually reach.
for “normal” stuff i think if you can set up good fuzzing this improves confidence in the code. you can use fuzzing on all levels meaning small ‘unit’ to ‘integration’ to whole gui application fuzzing. you can catch UAF, DF, leaks and all sorts of IO error conditions that are not properly handled, you can even catch races (of course empirically not mathematically). also a lot of logical bugs and edge cases. thats for no fear of users crashing your software.
for the advanced hackers attack your software part thats a hard question, probably i would start with reducing the attack surface, make the endpoint API really small, really simple, sanitize outside data with high discipline (with checks that are active in release fast). For IT SEC you always have to look at the whole picture so it really depends on what you do, yeah you can avoid some type of attacks (i.e using GC) but it comes with tradeoffs and there are lots of other ways you can get attacked (including physically). so choose your battles and since you already chose a language like zig that wants to give you maximum control to get optimal - start with fuzzing.
the best example for me in this case is sqlite. the testing they have set up is just impressive and the better the testing / correctness the less the programming language matters.
It’s interesting you bring up sqlite in this context. Richard Hipp recently gave a talk about the project, and one item that struck me was that they thought they had reached basically bug free software right before the explosion of AI vulnerability reports. But turns out they had not. And they are extremely rigorous in their testing.
So “tests will catch stuff” does not seem to cut it. It feels as if you have to always assume there are bugs despite your best efforts. Which is why outright disabling what could be your last line of defense application-wide (runtime checks) seems a bit unhinged and bold …
But yes, as usual the answer is always “it depends”. I didn’t have any particular projects in mind when I asked the question.
Is it not simply the fact that ghostty is significantly more TESTED than like … any C project ever?
The fact that ghostty isn’t getting popped with stack reference escape bugs (which Zig does nothing to help with), suggests that there is a qualitative difference.
I actually took the opposite view of Richard’s talk (which seemed to me to be closer to his view as well), which incidentally is also the philosophy approximately taken by the Zig core team: LLMs both reveal and help solve one further layer of bugs. These are (in mature software with a serious focus on testing and guarantees) not bugs that are solved by simply enabling or disabling safety checks.
My rendition of the levels would be as follows.
Software is in active (alpha) development. Using it often means contributing to the development process, in part because even the happy paths have bugs.
Beta software has eliminated most bugs in the happy path.
“Released” software has eliminated most bugs encountered by most users.
Software is near total test coverage and has begun to fuzz and/or to test on a branch / release binary level and not just on a source code correctness level
Software has achieved resilience against “centaur hacker” style attacks exploiting the compute power of LLMs with creativity.
So, to my mind the question is maybe not how paranoid are you, but how serious are you. I’d say if you’re aiming to reach level 4 or 5 with a piece of software that is performant, you should release in a release mode as soon as you are moving towards stage 3.
It really depends on what you are building. If your software is publicly exposed, you should probably be using ReleaseSafe as panicking is the better outcome. Zig 0.16 had ReleaseSafe slower than it should be due to the slow memset, but 0.17 should be much better.
There are exceptions, where you need the extra performance, and in those cases you simply need to do the work, have your own bounds checks, quadruple reviewed. You need to systematically avoid things that can lead to these kind if behaviors, for example, like the nginx, if you have a two step process to allocate and then fill the buffers, have a struct with all the inputs that you use in both passes and make sure the code is organized in a way to makes it visually clear both have the same logic. And then you make sure you have good test coverage, and that the test suite passes under valgrind/asan. And if you are using ReleaseFast, there might still be pieces of code where you opt in for Zig’s bounds checks anyway.
I’m going to say something very unpopular here, but LLMs are a great way to catch these bugs during development. If you have each of your changes reviewed by a competent LLM process, cases like the nginx RCE are trivial to catch for them, and nearly impossible to catch for human reviewers. This works way better than trying to scan your entire codebase for these kinds of bugs with LLMs. With single diff reviews, the LLMs can dig deeper into the logic of each piece and they are very good as pointing out the cases where you do something weird/stupid.