# Zig's Tokenizer and Pointer Provenance? **URL:** \ **Category:** Brainstorming\ **Created:** [March 23, 2025, 4:48pm UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221 "2025-03-23T16:48:39Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![marler8997](https://ziggit.dev/user_avatar/ziggit.dev/marler8997/32/77_2.png) [@marler8997](https://ziggit.dev/u/marler8997)\ **Post date:** [March 23, 2025, 4:48pm UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221/1 "2025-03-23T16:48:39Z")
I’ve used Zig’s tokenizer as a starting point for other tokenizers a few times. Based on what I’ve learned about pointer provenance (see Martin’s [talk](https://www.youtube.com/watch?v=kDwdgkCFbrQ)), it makes me wonder if changing this: ```zig pub const Tokenizer = struct { buffer: [:0]const u8, index: usize, pub fn next(self: *Tokenizer) Token { ... } }; ``` into something like this: ```zig pub fn next(buffer: [:0]const u8, index: *usize) Token { // ... } ``` would make things more “optimizer friendly”? Since `buffer` never changes but `index` does, putting them into the same struct/provenance might make it harder to optimize? Is this a useful way to think about things? Should we all be designing our types to avoid combining constant and mutating data? I created a benchmark here: [PerfTest · GitHub](https://gist.github.com/marler8997/28dccc0d10907398d877915a36a6e227) It compares the implementation from `std` with a copy that removes the `buffer` field from `Tokenizer` and adds it as an extra parameter to `next`. As expected the performance between the two is almost identical…however…on macOS and Windows the performance between runs seemed pretty consistent, consistent enough to say there might be a possible 1% improvement in performance? | Machine | Std Impl | Modified Impl | | --- | --- | --- | | M1 Air | 0.44 - 0.46 s | 0.43 - 0.45 s | | Win10 Desktop | 0.444 - 0.448 ms | 0.440 - 0.441 ms | On linux the variance was higher for some reason high enough that a 1% improvement might be undetectable. I haven’t tried looking at the assembly yet. ---
**Author:** ![cryptocode](https://ziggit.dev/user_avatar/ziggit.dev/cryptocode/32/817_2.png) [@cryptocode](https://ziggit.dev/u/cryptocode)\ **Post date:** [March 23, 2025, 5:58pm UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221/2 "2025-03-23T17:58:54Z")
To share an additional datapoint, I grabbed the perftest and ran on an M3 (multiple runs shows the same result): ```bash ⚡hyperfine "./perftest std" Benchmark 1: ./perftest std Time (mean ± σ): 42.5 ms ± 0.3 ms [User: 42.2 ms, System: 0.2 ms] Range (min … max): 41.9 ms … 43.6 ms 67 runs ⚡hyperfine "./perftest custom" Benchmark 1: ./perftest custom Time (mean ± σ): 43.3 ms ± 0.4 ms [User: 43.1 ms, System: 0.2 ms] Range (min … max): 42.7 ms … 45.0 ms 65 runs ``` Which seems to slightly favor std. Using a large input file like Sema.zig shows no meaningful difference. Compiled with zig trunk. Maybe the variance comes down to just system dependent changes to process data layout between the two implementations? ---
**Author:** ![permutationlock](https://ziggit.dev/user_avatar/ziggit.dev/permutationlock/32/9958_2.png) [@permutationlock](https://ziggit.dev/u/permutationlock)\ **Post date:** [March 23, 2025, 6:13pm UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221/3 "2025-03-23T18:13:43Z")
In Martin’s talk all of the examples where memory region escapes/provenance resulted in pessimization involved functions calls that were opaque to the compiler (i.e. loading a virtual function or crossing a translation unit boundary). The Zig tokenizer is compiled in one unit, and as far as I can see it doesn’t even make function calls except to lookup in a static string map, so at a glance this doesn’t look like a case where provenance issues would crop up. ---
**Author:** ![squeek502](https://ziggit.dev/user_avatar/ziggit.dev/squeek502/32/409_2.png) [@squeek502](https://ziggit.dev/u/squeek502)\ **Post date:** [March 23, 2025, 6:43pm UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221/4 "2025-03-23T18:43:52Z")
Use `hyperfine "./perftest std" "./perftest custom"` to get a comparison printed out as well. ---
**Author:** ![marler8997](https://ziggit.dev/user_avatar/ziggit.dev/marler8997/32/77_2.png) [@marler8997](https://ziggit.dev/u/marler8997)\ **Post date:** [March 23, 2025, 7:15pm UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221/5 "2025-03-23T19:15:39Z")
Thanks for sharing your data. I forgot I could use hyperfine on macOS and when I do so I’m actually seeing similar results as you on my M1, std seems to consistently perform slightly better. Not what I was expecting but very interesting nonetheless. Also strange I’m seeing the opposite on Windows. Unfortunately this little tangent only seems to have raised more questions then answers. ---
**Author:** ![squeek502](https://ziggit.dev/user_avatar/ziggit.dev/squeek502/32/409_2.png) [@squeek502](https://ziggit.dev/u/squeek502)\ **Post date:** [March 23, 2025, 7:23pm UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221/6 "2025-03-23T19:23:22Z")
If you haven’t seen it, this talk is something to keep in mind for this sort of stuff: [![](https://ziggit.dev/uploads/default/original/2X/0/05029cc0f0a762711216bf254a0e073bb0c6833e.jpeg ""Performance Matters" by Emery Berger") ](https://www.youtube.com/watch?v=r-TLSBdHe1A) ---
**Author:** ![cryptocode](https://ziggit.dev/user_avatar/ziggit.dev/cryptocode/32/817_2.png) [@cryptocode](https://ziggit.dev/u/cryptocode)\ **Post date:** [March 23, 2025, 7:26pm UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221/7 "2025-03-23T19:26:21Z")
Yeah that’s the one I had in mind, the username bit is so funny ---
**Author:** ![marler8997](https://ziggit.dev/user_avatar/ziggit.dev/marler8997/32/77_2.png) [@marler8997](https://ziggit.dev/u/marler8997)\ **Post date:** [March 24, 2025, 11:10am UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221/8 "2025-03-24T11:10:32Z")
Great video…should be “required reading (watching)” for every programmer 🙂 ---
**Author:** ![mnemnion](https://ziggit.dev/user_avatar/ziggit.dev/mnemnion/32/2478_2.png) [@mnemnion](https://ziggit.dev/u/mnemnion)\ **Post date:** [March 24, 2025, 6:28pm UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221/9 "2025-03-24T18:28:00Z")
As a companion to this video, I give [The Slow Winter](https://scholar.harvard.edu/files/mickens/files/theslowwinter.pdf) by the inimitable James Mickens my highest recommendation. Emery Berger explains the facts of life in this demon-haunted world. Mickens tells us where the demons came from. ---
**Author:** ![LucasSantos91](https://ziggit.dev/user_avatar/ziggit.dev/lucassantos91/32/440_2.png) [@LucasSantos91](https://ziggit.dev/u/LucasSantos91)\ **Post date:** [March 24, 2025, 6:54pm UTC](https://ziggit.dev/t/zigs-tokenizer-and-pointer-provenance/9221/10 "2025-03-24T18:54:05Z")
Compilers can analyze each field in a struct as if it were a separate argument, so it shouldn’t matter. This is called field sensitive pointer analysis. Obviously, this is more costly in terms of compilation time than tracking the whole struct, which is why GCC exposes an option to tune how intensely GCC should do this: max-fields-for-field-sensitive.