New Compaction-based Zig Tokenizer

I do have a similar project, inspired from your work. One function you optimize on all three compilers is the getKeyword(). Instead of strncmp for loop every keyword, replace it with a switch on keyword length, then memcmp the string.. It made GCC 14 go from 22 ms to 10.5 ms. TCC 0.9.27 went from around 35 ms to 22 ms. Clang 19 stayed the same at around 7.5 ms. All reading from Zig 0.14 src/Sema.zig

3 Likes